Python 3 - CSV and cx_Oracle

Question

I'm having some serious trouble working with the csv and cx_oracle module. I want to read a csv file, that is saved in UTF-8 (I checked it by saving it with Notepad in UTF-8). I can read everything fine now (before I saved it as UTF-8 it didn't). This is my code to read the csv-file:

 with open(file, 'rt', encoding='utf-8') as csvfile:
    csvinput = csv.reader(csvfile, delimiter = ',', quotechar = '"')
    for row in csvinput:
        data.append(row)

This saves everything to a 2D array. Whenever I want to insert something into the database, I make a preparedstatement, and load the text into it as such:

data = [lastname, firstname]
cursor = cx_Oracle.Cursor(connection)
cursor.prepare("SELECT * FROM PRIVATE WHERE NAME = :1 AND FIRSTNAME = :2")
cursor.execute(None, data)
res = cursor.fetchall()
cursor.close()

It gives me tons of errors like:

UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 1: ordinal not in range(128)

I tried reading up on the whole thing, but I got rather confused with the unicode thing as I don't really know where I should use what and why... Any help is appreciated. TLDR I get encoding errors whilst trying to execute prepared statements

Answer 1

You are trying to insert Unicode values into a VARCHAR2 column, which can only handle encoded byte strings.

cx_Oracle is trying to encode your Unicode values for you to fit the column type, and does so with the default codec for your connection.

Either encode your values to a suitable encoding manually or make your columns use NVARCHAR2 instead.

The latter has the added advantage that column lengths are expressed in characters, not bytes; UTF-8 data can use up to 4 bytes per character, so a VARCHAR2(1000) column could, in a worst-case scenario, fit only 250 actual characters.