Charsets in general, with specific questions regards Zend DB - incorrect display of data

Question

I'm having trouble in my Zend Framework App displaying the correct characters.

EG/ rue de l'Odéon

Displays as:

rue de l'Od�on

When displaying information through a webservice and also by displaying it through the applications view.

I have tryed the following things:

1) Set the character encoding in the view.

$view->setEncoding('UTF-8');

2) Ensured that col / table encoding on the underlying database is UTF8 (mysql db)

ALTER TABLE tablename CONVERT TO CHARACTER SET utf8;

3) Passed the charset varaible to the Zend_Db::Factory method when creating my Db connection. This is the config section of the object I obtained through the registry.

 [_config:protected] => Array
    (
        [host] => localhost
        [username] => root
        [password] => ---------------
        [dbname] => db_staging
        [charset] => utf8
        [options] => Array
            (
                [caseFolding] => 0
                [autoQuoteIdentifiers] => 1
            )

Answer 1

A few questions below. I know some of these sound completely silly, but when I get stuck like this I automatically assume it's something so obvious I will spend days kicking myself! :) First of all, I assume you are using ZF 1.8 given the charset option above? If not (or even if the list below doesn't solve it) switch back to the older method to see if that solves anything first.

1. Can you confirm table collation is now utf8_general_ci, as are all text fields? The command above doesn't mention a collation so I assume this was set as the default collation (it's a default when missing and the charset is utf8).
2. Now, resend that command but also set the collation to utf8_unicode_ci (slower but more accurate than the general collation - performance will take a tiny hit but it might help with an existing table of data).
3. Have you re-input the data just to be sure? Even type it into a known UTF-8 editor just to double check you definitely have copied utf8 characters; and
4. Are you 100% certain the source text is UTF-8 and not something like ISO-8859-1 or Windows 1252?
5. Make sure the View encoding is set as early as possible (and the new View is added to ViewRenderer helper).
6. What mysql driver is used? Mysqli or PDO? Have you used the appropriate settings for your option?
7. Is there any View overriding of character encoding? Encoding is taken from the Content-Type header, then any XML declaration (e.g. XHTML), and finally from a meta http-equiv tag for Content-Type. This is also the precedence order - if an earlier declaration disagrees with UTF-8 the page may not use the one you intended. You can reset the header by setting a custom Response object on the front controller with the correct header set - usually "Content-Type: text/html; charset=UTF-8".
8. To detect encoding if incorrectly detected by browser, you can also manually debug by resetting the browsers encoding for the page, e.g. View > Character Encoding in Firefox. Check is ISO or Windows encoding display it better.
9. Try wrapping the text output in template in utf8decode() - maybe something is double encoding it? Weirder things have happened...

My UTF-8 crisis checklist ;). Let me know if anything listed helps. I think it covers most of the items I can think of.

Answer 2

In addition to Pádraic's excellent list, have a look at http://akrabat.com/2009/03/18/utf8-php-and-mysql/