Using UTF and Hindi in CakePHP and MySQL

Question

I've create a form that contains Hindi (UTF-8) data which i want to store in MySQL table. The columns corresponding to UTF data has collation value set to utf_general_ci.

I've successfully stored the data in table but when I'm executing a select-where query, it doesn't returns the data. Here is my query:

SELECT Birth.sno, Birth.bookingnumber, Birth.birth_date, Birth.baby_gender, Birth.baby_name, Birth.baby_father_name, Birth.baby_father_address, Birth.baby_mother_name, Birth.birth_place, Birth.place_type, Birth.applicant_name, Birth.applicant_address, Birth.registration_number, Birth.registration_date, Birth.registration_ward, Birth.registration_city_village, Birth.registration_district, Birth.remark, Birth.mother_place_name, Birth.mother_place_type, Birth.mother_place_district, Birth.mother_place_state, Birth.person_religion, Birth.father_education, Birth.mother_education, Birth.father_occupation, Birth.mother_occupation, Birth.mother_age_at_marriage, Birth.mother_age_at_birth, Birth.count_of_mother_child, Birth.birth_by, Birth.birth_method, Birth.mother_weight_at_birth, Birth.pregnancy_duration, Birth.date_of_issue FROM np.births AS Birth WHERE Birth.baby_name = 'd' AND Birth.baby_father_name = 'e' AND Birth.baby_mother_name = 'f' AND Birth.baby_father_address = 'g' AND Birth.person_religion = 'हिंदू' AND Birth.baby_gender = 'पुरुष'

The name of the database is np and name of the table is births The above query was printed in the log file. I tried to copy and paste the same query in HeidiSQL (front end for MySQL) but its not running. However, if I remove the following part: ** AND Birth.person_religion = 'हिंदू' AND Birth.baby_gender = 'पुरुष'**, the query works fine.

How can I resolve this issue?

Answer 1

This looks like a case when your MySQL client and your MySQL server do not "talk" the same encoding.

There are 3 places where you need to take care of your encoding.

The Web Form (what the users sees) -> Your Web Application (CakePHP) -> Your Database Server (MySQL)

One of those three is NOT using the same encoding as the others. So by the time:

"'हिंदू'" and "'पुरुष'" get to your database they will be something totally different that will not be found in the database.

So, make sure that in your default.ctp file you have set your encoding:

echo $this->Html->charset(); //this will result in a UTF-8 encoding of the page.

Look at the source code of your web page (where I guess you have a search/filter form).

At the top you should see:

<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

Then look for the code generated for your search/filter form. You should see:

<form id="your_form_id" accept-charset="utf-8" method="post" action="/your/action/">

The important part is that it that "utf-8" that MUST show up in those places.

Next, look into your database.php file and make sure this line:

'encoding' => 'utf8', is NOT commented out!

Finally, with a client that you are sure supports UTF-8 (probably HeidiSQL) have a look at your data table np.births and make sure that what data you have there actually makes sense! It's possible it got mangled because of the discrepancies in encoding before.

Once the data makes sense in the database you should be good to go!

IF this does not do it you, you'll have to read and thoroughly understand this article. Only then you will be able to locate where the problem is and get your encodings in sync. (Obviously your PHP source files should be UTF-8 encoded as well...)