Oracle OCI changeing invalid UTF8 characters to U+FFFD

Question

I am writing a C++ data conversion program which is copying data from an ODBC data source into an Oracle database. Have chosen C++ (with array operations) due to the very high volume of data to move (billions of rows).

Now the text columns are "supposed" to be UTF-8, but this is not always the case. When its not I still want to copy the invalid raw bytes into Oracle. We will clean them up later. The column is a simple VARCHAR2(100), so 100 bytes long. But Oracle appears to be attempting some sort of UTF-8 parsing/processing on the data.

For example the following string (has been truncated to 100 bytes, thus invalid):

Hex Bytes: 46 46 54 F0 9F 98 84 F0 9F 98 88 F0 9F 98 94 F0 9F 98 85 F0 9F 98 90 F0 9F 98 88 F0 9F 98 94 F0 9F 98 88 F0 9F 98 85 F0 9F 98 94 F0 9F 98 86 F0 9F 98 94 F0 9F 98 85 F0 9F 98 90 F0 9F 98 90 F0 9F 98 86 F0 9F 98 90 F0 9F 98 90 F0 9F 98 87 F0 9F 98 90 F0 9F 98 92 F0 9F 98 88 F0 9F 98 9A F0 9F 98 88 F0

http://tinyurl.com/nhhkf62

Is actually being inserted into the database as:

Hex Bytes: 46 46 54 EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD EF BF BD

http://tinyurl.com/orkv6z6

Which is basically the leading 3 ascii chars followed by the UTF-8 encoding of U+FFFD for each of the subsequent bytes.

Other details:

Oracle Version: 11g Enterprise Edition Release 11.2.0.1.0
Oracle Client: oracle-instantclient11.2-basic-11.2.0.3.0-1
Oracle OCI rpm: oracle-instantclient11.2-devel-11.2.0.3.0-1
Environment: LANG=en_US.UTF-8
Environment: NLS_CHARACTERSET=AMERICAN_AMERICA.UTF8
Environment: NLS_LANG=AMERICAN.UTF8

So does anyone know why Oracle and/or OCI is modifying this data? And is there a way to stop it from happenning?

Thanks

Answer 1

NLS_LANG is most important for implicit character conversion. I think it should be NLS_LANG=AMERICAN_AMERICA.UTF8 instead of NLS_LANG=AMERICAN.UTF8

What is your database character set?