Error while writing unicode data to file

Question

I am trying to write unicode data(The actual data contains german characters) to a file but I am getting error:

Traceback (most recent call last):
  File "C:\Python27\extract_osm_road_nw.py", line 76, in <module>
    file.write(str(list_way_id[index][2][i][1]))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xdf' in position 12: ordinal not in range(128)

The code is:

## writing the data in a file
## The data format is:
## A list of: [tuple(way ref id,list[tuple(node id, lat, long)],list[tuple(key,value)]),.....]
## For example: [(u'72439830', [(u'298094414', u'52.4626304', u'10.5579578'), (u'860126050', u'52.4626762', u'10.5576574')], [(u'name', u'General-BeckStra\xdfe')]),.....]

with codecs.open("extracted_osm_file.csv", "w", encoding="utf-8") as file:
    for index in range(len(list_way_id)):
        file.write("Way ID Ref No: ")
        file.write(str(list_way_id[index][0]))## points to the way id ref no
        file.write("\n")
        file.write("Node reference id, latitude, longitude: ")
        file.write("\n")
        for i in range(len(list_way_id[index][1])):
            file.write(str(list_way_id[index][1][i][0]))
            file.write(",")
            file.write(str(list_way_id[index][1][i][1]))
            file.write(",")
            file.write(str(list_way_id[index][1][i][2]))
            file.write("\n")
        for i in range(len(list_way_id[index][2])):
            file.write(str(list_way_id[index][2][i][0]))
            file.write(",")
            file.write(str(list_way_id[index][2][i][1]))
            file.write("\n")
    file.close()

Answer 1

Remove the calls to str(). These try to convert the unicode data you have to bytecode, using the default ASCII encoding.

You probably want to use the csv module here and save yourself the grief of having to write all those commas, but if you cannot, at least use loops properly:

with codecs.open("extracted_osm_file.csv", "w", encoding="utf-8") as fileobj:
    for line in list_way_id:
        fileobj.write(u'Way ID Ref No: {}\n'.format(line[0]))
        fileobj.write(u'Node reference id, latitude, longitude: \n')
        fileobj.write(u'\n'.join([u','.join(subentry)
            for entry in line[1:] for subentry in entry]))

which, for your partial example, would write:

Way ID Ref No: 72439830
Node reference id, latitude, longitude: 
298094414,52.4626304,10.5579578
860126050,52.4626762,10.5576574
name,General-BeckStra\xc3\x9fe