Extended ASCII characters show up as junk in MySQL db is inserted through perl

Question

I have a MySQL 'articles' table and I am trying to make the following insert using SQLyog.

insert into articles (id,title) values (2356606,'Jérôme_Lejeune');

This works fine and the data shows fine when I do a select query.

The problem is that when I do the same insert query through my perl script, the name shows up with some junk characters in place of é and ô in the database. I need to know how to properly store the name through my script. The part of code that does the insert is like this.

$sql_insert = "insert into articles (id,title) values (?,?)";
$sth_insert = $dbh->prepare($sql_insert);
$sth_insert->execute($id,$title);

$id and $title have the correct required data which I have checked by print before I am inserting them. Please assist.

Answer 1

There are few things to follow.

First you have to make sure, that Perl understands that data which is moving between your program and DB is encoded as UTF-8 (i expect your databases and tables are set properly). For this you need to say it loud out on connecting to database, like this:

 my($dbh) = DBI->connect(
     'dbi:mysql:test',
     'user',
     'password',
     {
         mysql_enable_utf8 => 1,
     }      
);

Next, you need send data to output and you must set it to decaode data as UTF-8. For this i like pretty good module:

use utf8::all;

But this module is not in core, so you may want to set it with binmode yourself too:

binmode STDIN, ":utf8"; 
binmode STDOUT, ":utf8"; 

And if you deal with webpages, you have to make sure, that browser understoods that you are sending your data encoded as UTF-8. For that you should make sure your HTTP-headers include encoding:

Content-Type: text/html; charset=utf-8;

and set it with HTML META-tag too:

<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

Now you should get your road covered.

Answer 2

You have opened up the character encoding can of worms, and you have a lot to learn before you will solve this problem and have it stay solved.

You are probably already used to thinking of how a character of text can be encoded as a string of bits. Under the ASCII encoding, for example, the 8-bit string 01000001 (65) is used to indicate the A character. When you start to think about how many different languages there are and how many different kinds of characters there are, you quickly realize that an 8-bit encoding is not going to get you very far. So a number of other character encodings have proliferated. Some of the most popular are latin1 (ISO-8859-1) and UTF-8. Both of these encodings can render the é and ô characters, but they use quite different bit strings to represent them. As you write to a file (or to the terminal) or add a row to a database, Perl and MySQL have a notion of what the character encoding of the output stream is. An encoding is also used when you read data. If you don't know what this encoding is, then it doesn't make any sense to say that the data looks good/looks bad when you store it and retrieve it.

Perl and MySQL can, with the right settings, handle both of these encodings and several more. Which encoding you choose to use is not as important as making sure that all the pieces of your application are using the same encoding. But you should choose an encoding that

1. can encode all of the characters you will need (for this problem, you mention é and ô, but will there be others? what about in the future?)
2. is supported by all the pieces of your application (front-end, database, back-end)

Here's some suggested reading to get you headed in the right direction:

• The Encode module for Perl
• character sets in MySQL
• (others should feel free to recommend additional links)

I can't speak to MySQL so much, but character encoding support in Perl is rapidly evolving (which isn't to say that it ain't damn good). The latest versions of Perl will have the best support (for the most obscure character sets) and the best features (for example, regular expressions and character classes) for characters beyond ASCII.