Unicode: Python / lxml file output not as expected (print vs write)

Question

I'm parsing an xml file using the code below:

import lxml

file_name = input('Enter the file name, including .xml extension: ')
print('Parsing ' + file_name)

from lxml import etree

parser = lxml.etree.XMLParser()


tree = lxml.etree.parse(file_name, parser)
root = tree.getroot()

nsmap = {'xmlns': 'urn:tva:metadata:2010'} 


with open(file_name+'.log', 'w', encoding='utf-8') as f:
    for info in root.xpath('//xmlns:ProgramInformation', namespaces=nsmap):
       crid = (info.get('programId'))
       titlex = (info.find('.//xmlns:Title', namespaces=nsmap))
       title = (titlex.text if titlex != None else 'Missing')
       synopsis1x = (info.find('.//xmlns:Synopsis[1]', namespaces=nsmap))             
       synopsis1 = (synopsis1x.text if synopsis1x != None else 'Missing')               
       synopsis1 = synopsis1.replace('\r','').replace('\n','')
       f.write('{}|{}|{}\n'.format(crid, title, synopsis1))    

Let take an example title of 'Přešité bydlení'. If I print the title whilst parsing the file, it comes out as expected. When I write it out however, it displays as 'PÅ™eÅ¡ité bydlení'.

I understand that this is do to with encoding (as I was able to change the print command to use UTF-8, and 'corrupt' the output), but I couldn't get the written output to print as I desired. I had a look at the codecs library, but couldn't wasn't successful. Having 'encoding = "utf-8"' in the XML Parser line didn't make any difference.

How can I configure the written output to be human readable?

Answer 1

I had all sorts of troubles with this before. But the solution is rather simple. There is a chapter on how to read and write in unicode to a file in the documentation. This Python talk is also very enlightening to understand the issue. Unicode can be a pain. It gets a lot easier if you start using python 3 though.

import codecs
f = codecs.open('test', encoding='utf-8', mode='w+')
f.write(u'\u4500 blah blah blah\n')
f.seek(0)
print repr(f.readline()[:1])
f.close()

Answer 2

Your code looks ok, so I reckon your input is duff. Assuming you're viewing your output file with a UTF-8 viewer or shell then I suspect that the encoding in the <?xml doesn't match the actual encoding.

This would explain why printing works but not writing to a file. If your shell/IDE is set to "ISO-8859-2" and your input XML is also "ISO-8859-2" then printing is pushing out the raw encoding.