UTF-8 htmlentities and fgetcsv

Question

I am having issues converting my special characters to htmlentities after importing my csv file.

Here's the revelant code:

setlocale(LC_ALL, 'fr_FR.utf8');

if (empty($errors) && ($handle = fopen($_FILES["file"]["tmp_name"], "r")) !== FALSE) {
    $data = array();
    while (($rawdata = fgetcsv($handle, 0, $_POST["delimiter"])) !== FALSE) {
         for ($i=0; $i < count($rawdata); $i++) {
              $data[$i][] = htmlentities(trim($rawdata[$i]), ENT_QUOTES, "UTF-8");
         }
    }
    fclose($handle);
}

What happens though, is that any cells with a special character in it (such as ™) simply get removed / return as empty.

I'm using PHP version 5.3.13

I have tried setting my locale and tried putenv, but this doesn't change anything. I have also tried setting my machine's locale settings before making the csv. The csv itself is created from an Excel file.

I have checked my csv encoding, and it seems correctly to be UTF-8 without BOM (checked in Notepad++). mb_detect_encoding() also returns UTF-8.

When I change to ENT_IGNORE, it simply strips the TM symbol from my string. I have tried different encoding types such as ISO-8859-15 with no avail.

str_replace("™", "%99", $row) just ignores the TM symbols and leaves them how they were.

I've found that a lot of people have issues with fgetcsv() and encoding / special characters, and most of them refer to using a different method such as fgets(). Unfortunately I haven't been able to get those other methods to work either because I cannot explode on newline since some of the cells may include newlines in their content. I will accept a different method as answer as well if I can get it to work.

Answer 1

Using iconv() on my rawdata in the for loop solved my issue:

$data[$i][] = htmlentities(iconv("cp1252", "utf-8", trim($rawdata[$i])), ENT_IGNORE, "UTF-8");

Thank you @Leigh, Wrikken and DaveRando from the PHP chat ;)