Python converting latin1 to UTF8 [duplicate]

Question

This question already has answers here:

Python: Converting from ISO-8859-1/latin1 to UTF-8 (5 answers)

Closed 11 months ago.

In Python 2.7, how do you convert a latin1 string to UTF-8.

For example, I'm trying to convert é to utf-8.

>>> "é"
'\xe9'
>>> u"é"
u'\xe9'
>>> u"é".encode('utf-8')
'\xc3\xa9'
>>> print u"é".encode('utf-8')
é

The letter is é which is LATIN SMALL LETTER E WITH ACUTE (U+00E9) The UTF-8 byte encoding for is: c3a9
The latin byte encoding is: e9

How do I get the UTF-8 encoded version of a latin string? Could someone give an example of how to convert the é?

Answer 1

To decode a byte sequence from latin 1 to Unicode, use the .decode() method:

>>> '\xe9'.decode('latin1')
u'\xe9'

Python uses \xab escapes for unicode codepoints below \u00ff.

>>> '\xe9'.decode('latin1') == u'\u00e9'
True

The above Latin-1 character can be encoded to UTF-8 as:

>>> '\xe9'.decode('latin1').encode('utf8')
'\xc3\xa9'

Answer 2

>>> u"é".encode('utf-8')
'\xc3\xa9'

You've got a UTF-8 encoded byte sequence. Don't try to print encoded bytes directly. To print them you need to decode the encoded bytes back into a Unicode string.

>>> u"é".encode('utf-8').decode('utf-8')
u'\xe9'
>>> print u"é".encode('utf-8').decode('utf-8')
é

Notice that encoding and decoding are opposite operations which effectively cancel out. You end up with the original u"é" string back, although Python prints it as the equivalent u'\xe9'.

>>> u"é" == u'\xe9'
True

Answer 3

concept = concept.encode('ascii', 'ignore') concept = MySQLdb.escape_string(concept.decode('latin1').encode('utf8').rstrip())

I do this, I am not sure if that is a good approach but it works everytime !!