Convert or strip out “illegal” Unicode characters

Question

I've got a database in MSSQL that I'm porting to SQLite/Django. I'm using pymssql to connect to the database and save a text field to the local SQLite database.

However for some characters, it explodes. I get complaints like this:

UnicodeDecodeError: 'ascii' codec can't decode byte 0x97 in position 1916: ordinal not in range(128)

Is there some way I can convert the chars to proper unicode versions? Or strip them out?

Answer 1

Once you have the string of bytes s, instead of using it as a unicode obj directly, convert it explicitly with the right codec, e.g.:

u = s.decode('latin-1')

and use u instead of s in the code that follows this point (presumably the part that writes to sqlite). That's assuming latin-1 is the encoding that was used to make the byte string originally -- it's impossible for us to guess, so try to find out;-).

As a general rule, I suggest: don't process in your applications any text as encoded byte strings -- decode them to unicode objects right after input, and, if necessary, encode them back to byte strings right before output.

Answer 2

When you decode, just pass 'ignore' to strip those characters

there is some more way of stripping / converting those are

'replace': replace malformed data with a suitable replacement marker, such as '?' or '\ufffd' 

'ignore': ignore malformed data and continue without further notice 

'backslashreplace': replace with backslashed escape sequences (for encoding only) 

Test

>>> "abcd\x97".decode("ascii")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0x97 in position 4: ordinal not in range(128)
>>>
>>> "abcd\x97".decode("ascii","ignore")
u'abcd'