how to detect WebPage charset,and get page content?

Question

i use follows code to get page content:

URL url=new URL("http://www.google.com.hk/intl/zh-CN/privacy.html");
BufferedReader reader = new BufferedReader(new InputStreamReader(url.openConnection().getInputStream()));
for(String line=reader.readLine();line!=null;line=reader.readLine()){
    System.out.println(line);
}
reader.close();

page: http://www.google.com.hk/intl/zh-CN/privacy.html charset is "UTF-8",but my system default charset is "GBK",so, these code can't type right.

i know ,i can write a charsetname in InputStreamReader constructor:

new InputStreamReader(url.openConnection().getInputStream(),"UTF-8")

it's will be ok, but i want to know:

how to detect charset,and get page content? (not send two requests better)

any java library can do this? (get webpage content,and don't need set charsetname)

thanks for help :)

Answer 1

There is really no easy way to detect the proper charset. You can hope that the web page you are interested in declares the charset using a <meta charset="utf-8"> tag. When you detect that tag you could switch charset of your parsing.

There are also some libraries that make an effort to detect the charset, for example http://jchardet.sourceforge.net/.