Get Python 2.7's 'json' to not throw an exception when it encounters random byte strings

Question

Trying to serialize a dict object into a json string using Python 2.7's json (ie: import json).

Example:
json.dumps({
    'property1':        'A normal string',
    'pickled_property': \u0002]qu0000U\u0012
})

The object has some byte strings in it that are "pickled" data using cPickle, so for json's purposes, they are basically random byte strings. I was using django.utils's simplejson and this worked fine. But I recently switched to Python 2.7 on google app engine and they don't seem to have simplejson available anymore.

Now that I am using json, it throws an exception when it encounters bytes that aren't part of UTF-8. The error that I'm getting is:

UnicodeDecodeError: 'utf8' codec can't decode byte 0x80 in position 0: invalid start byte

It would be nice if it printed out a string of the character codes like the debugging might do, ie: \u0002]q\u0000U\u001201. But I really don't much care how it handles this data just as long as it doesn't throw an exception and continues serializing the information that it does recognize.

How can I make this happen?

Thanks!

Answer 1

The JSON spec defines strings in terms of unicode characters. For this reason, the json module assumes that any str instance it receives holds encoded unicode text. It will try UTF-8 as its default encoding, which causes trouble when you have a string like the output of pickle.dumps which may not be a valid UTF-8 sequence.

Fortunately, fixing the issue is easy. You simply need to tell the json.dumps function what encoding to use instead of UTF-8. The following will work, even though my_bytestring is not valid UTF-8 text:

import json, cPickle as pickle

my_data = ["some data", 1, 2, 3, 4]
my_bytestring = pickle.dumps(my_data, pickle.HIGHEST_PROTOCOL)
json_data = json.dumps(my_bytestring, encoding="latin-1")

I believe that any 8-bit encoding will work in place of the latin-1 used here (just be sure to use the same one for decoding later).

When you want to unpickle the JSON encoded data, you'll need to make a call to unicode.decode, since json.loads always returns encoded strings as unicode instances. So, to get the my_data list back out of json_data above, you'd need this code:

my_unicode_data = json.loads(json_data)
my_new_bytestring = my_unicode_data.encode("latin-1")  # equal to my_bytestring
my_new_data = pickle.loads(my_new_bytestring)          # equal to my_data