Delphi, dbExpress and UTF8

Question

We've just upgraded our MySQL server from 5.0 to 5.1.

The remaining task is to finally sort out our Unicode support, which has been troubling us since Delphi 2010. We are now using XE5.

The new database has been set up with default collation as utf8-general-ci. The tables have been migrated with their original collation as latin1-swedish-ci.

At the moment all our applications read with TSQLConnection.ServerCharSet = "" as setting the read connections to utf8 had unexpected results.

We write with connections set as TSQLConnection.ServerCharSet = "utf8"

I have now made a local copy of the new schema and have converted all the tables to utf8-general-ci - and this looks to have gone smoothly.

However, when I test on the new server it still is not smooth:

• Setting the read connection to utf8 delivers results in some form of Kanji.
• Setting the read connection to utf8-general-ci or "" reads correctly.
• Setting the write connection to "" results in garbage when writing "£" or "€" chars
• Setting the write connection to utf8 writes correctly

I suspect that the utf8-general-ci is not being recognised as an option by dbExpress so is being ignored

I also suspect that the SQLConnection.Locale setting may be an issue - currently this is set to 0000, but I've tried setting it to LOCALE_SYSTEM_DEFAULT (2048) with no success...

Is there any way we can avoid having 2 separate TSQLConnections for Read / Write routines?

edit

Based on suggestions by Arioch 'The I've ran a test with different string column types. We've been using Text/Tinytext fields and that's where we've been seeing the problems.

The problem does not occur when using VarChar column types.

We'd like a suggestion that doesn't require us to use a third party driver/connection as that move does not fit with our current development strategy

Answer 1

The Final Comment from Arioch 'The lead me to find this question:

With Delphi Xe2 and MySQL 5.1, how return UTF8 string from query?

Which in turn lead to this Embarcadero page:

dbExpress Data Type Mapping for Supported Databases

Wherein the answer is found:

text TDBXDataTypes.AnsiStringType

i.e. Read AsAnsiString rather than AsString