UTF-8 weird behaviour between HTML/PHP forms and MySQL (Hindi)

Question

I have all my database/tables and columns set to UTF-8_general_ci collation set.

Conditions that I Faced :-

1. When I insert hindi data manually by phpmyadmin, I can see the the hindi characters in phpmyadmin, while question marks when seen on webpage generated by PHP
2. In the same table when I insert data by HTML/PHP Forms I see some unrecognizable words in english something like cc2faa;(something like this) and Correct Hindi on Webpage.
3. For the large data we have a script that reads from txt files and insert the data in the table in this , I see characters like जाना in phpmyadmin but Hindi On webpage.

Now the main problem is :-

Data has gone under changes online by forms and now I need this data to export to excel and give to the client but I am getting जाà¤ in excel instead of Hindi Characters.

Note :-

• All English characters are working fine and as it is everywhere.
• My CHARACTER SET is utf8 For all tables.
• I tried to change the collation to UTF-8_bin but that too doesn't helped me in anyway.
• Encoding on the browser is UTF-8, and I have already sent the headers for UTF-8 encoding.

I have seen many posts about utf8 problem but no one seem to have this weird different behavior problem.

Please Do I have any rescue from this?? Or finally have to give the PHP reports of the data??

Please help!!

Answer 1

1. When I insert hindi data manually by phpmyadmin, I can see the the hindi characters in phpmyadmin, while question marks when seen on webpage generated by PHP

PHP probably generates the question marks because the encoding of the database connection is not utf-8. How to fix this depends on the database library you use; if you use MySQLi use mysqli_set_charset('utf8'), if PDO you add charset=utf8 to the DSN...

1. In the same table when I insert data by HTML/PHP Forms I see some unrecognizable words in english something like cc2faa;(something like this) and Correct Hindi on Webpage.
2. For the large data we have a script that reads from txt files and insert the data in the table in this , I see characters like जाना in phpmyadmin but Hindi On webpage.

These are likely caused by the same problem as above: the PHP forms and the script connect to the database using the default encoding, probably latin1. Then they insert utf-8 encoded text, but since MySQL thinks you are using latin1, it encodes the text into utf-8 again, and inserts this doubly encoded text into the table.

So: PHP sends "जाना" to MySQL telling it's latin1, and MySQL goes and converts it to utf-8, resulting in "जाना". Later PHP asks MySQL return the value, and since the connection is again using latin1, MySQL takes "जाना" and decodes it to latin1. Then PHP pretends that this latin1 string is actually utf-8 and displays "जाना".

Again, the solution is setting the encoding of the connection to utf-8. And this depends on what you use to access the database.

Answer 2

If you need to export your data as Excel file, use the PHP class php-export-data by Eli Dickinson, http://github.com/elidickinson/php-export-data. It is pretty nifty and so far I had have no problems exporting weird character sets with it.