Unicode in Python on Google App Engine

Question

I need to make a POST request in which the data might be non-ascii (chinese, japanese characters). I need to convert the input to unicode and encode with utf-8. Here's how I did it:

foo = unicode(self.request.get('foo'), 'utf-8') #convert to unicode
foo = foo.encode('utf-8')                       #encode with utf-8
data = {'foo': foo}
payload = urllib.urlencode(data)

However, I keep getting this error in my logs:

TypeError: decoding Unicode is not supported

Answer 1

Unicode can't be decoded because it's already unicode.

Try this instead:

if isinstance(var, str):
    var = unicode(var, 'utf-8')
else:
    var = unicode(var)

Answer 2

Ok some comments:

 foo = unicode(self.request.get('foo'), 'utf-8') #convert to unicode

Don't call it "convert". Call it "decode", it makes it clearer.

 foo = foo.encode('utf-8')                       #encode with utf-8

But why? You just decoded it from UTF8, why are you encoding it back? You can just as well do:

 foo = self.request.get('foo')

That's equivalent to the above two lines.

To lessen your confusion on Unicode, read this: http://www.joelonsoftware.com/articles/Unicode.html