decode json encoded as GB2312

Question

Via a GET request, I pull json from the Google geocode API:

import urllib, urllib2

url = "http://maps.googleapis.com/maps/api/geocode/json"
params = {'address': 'ivory coast', 'sensor': 'false'}
request = urllib2.Request(url + "?" + urllib.urlencode(params))
response = urllib2.urlopen(request)
st = response.read()

What comes out looks like:

{
   "results" : [
      {
         "address_components" : [
            {
               "long_name" : "Côte d'Ivoire",
               "short_name" : "CI",
               "types" : [ "country", "political" ]
            }
         ],
         "formatted_address" : "Côte d'Ivoire",
         "geometry" : { ... # rest snipped

As you see, the country name has some encoding issues. I tried to guess the encoding like this:

import chardet
encoding = chardet.detect(st)
print "String is encoded in {0} (with {1}% confidence).".format(encoding['encoding'], encoding['confidence']*100)

Which returns:

String is encoded in GB2312 (with 99.0% confidence).

What I would like to know is how I can convert this into a dictionary with an encoding where the ô (o with circumflex) is properly displayed.

I tried:

st = st.decode(encoding['encoding']).encode('utf-8')

But then I get:

{
   "results" : [
      {
         "address_components" : [
            {
               "long_name" : "C么te d'Ivoire",
               "short_name" : "CI",
               "types" : [ "country", "political" ]
            }
         ],
         "formatted_address" : "C么te d'Ivoire",
         "geometry" : { ... # rest snipped

Answer 1

The google api results are always encoded in UTF-8, you can even read this manually from their HTTP Content-Type header:

Answer 2

Once you've (properly) decoded it, don't re-encode it; json can work with unicode perfectly well.

>>> json.loads(u"[\"C\xf4te d'Ivoire\"]")
[u"C\xf4te d'Ivoire"]