Unicode Characters issue in Cakephp and Mysql

Question

I have a site implemented in Cakephp 2 years before for my client. At that time i was not aware about the site will be used world wide. Due to different country special characters have been used in the site. At that time cakephp utf8 option was not enabled and characters are saved in encoded form like ?? in database.

Now when ever we try to download the CSV these characters created problem and not appeared correctly in CSV. I have tried a lot to resolve this but did not succeed.

Please help me how to resolve this.

Answer 1

You must ensure 3 things-

1. enable `'encoding' => 'utf8',` on database settings at `app/Config/database.php`
2. Table column Collation must be set to 'utf8_general_ci' or `utf8_unicode_ci`
3. Html page character set must set as `utf-8`

Answer 2

Use the Below Query to get the Current Collation information for your Tables,

select TABLE_NAME,TABLE_TYPE,ENGINE,TABLE_COLLATION from information_schema.TABLES where TABLE_SCHEMA like 'YOURDATABASENAME';

The Column 'TABLE_COLLATION' will give you the collation info. If it's set to utf8, then almost all the characters could be saved in your DB and can be retrieved back. The issue you currently facing could be because of PHP or Browser encoding problems. But if your DB has different collation, which doesn't supports all characters, then the data saved in that DB is probably lost. It's almost impossible to identify encoding type and retrieve it back.

For future cases, you have two choices,

You could set UTF-8 as your DB Collation, but if you have indexed your string data, then for each and every char, MySQL process will hold 4 Bytes, even though UTF-8 is variable length encoding.So this will possibly increase your Memory usage.

Or

You could set latin1 as your DB Collation and you have to url-encode the characters from UTF-8 to latin and save them in DB. This will decrease your memory usage, but you would have the over head of Encoding/Decoding. If English is the major language in your DB, i would say you can go for this.

It depends on the Language you need to save in your DB and when showing them back in browser, the Browser must have set the supported encoding. In your case, if you are downloading them as CSV,it depends on the encoding format set for the file.