Exporting CSV in french language shows junk charcters

Question

I am having a problem in exporting a csv file using au.com.bytecode.opencsv.CSVWriter. I did something like:

File file = File.createTempFile("UserDetails_", ".csv"); CSVWriter writer = new CSVWriter(new OutputStreamWriter( new FileOutputStream(file), "UTF-8"), ',');

and then when I exporting the .csv file, it shows the junk characters for french letters.[Data to be saved in the .csv are french characters]. But previously I was doing something like:

CSVWriter writer = new CSVWriter(new FileWriter(file));, then it was perfectly showing all french characters in Windows environment, but in Prod environment[Linux] it was showing junks. So I thought to use the Character set UTF-8 for the file format to be exported.

How can I get rid of the problem? Please Suggest!! Thanks in advance!

Answer 1

Hypothesis: you use Excel to open your CSVs under Windows.

Unfortunately for you, Excel is crap at reading UTF-8. Even though it should not be required, Excel expects to have a byte order mark at the beginning of the CSV if it uses any UTF-* encoding, otherwise it will try and read it using Windows 1252!

Solution? Errr... Don't use Excel?

Anyway, with your old way:

CSVWriter writer = new CSVWriter(new FileWriter(file));

this would use the JVM's default encoding; this is windows-1252 under Windows and UTF-8 under Linux.

Note that Apache's commons-io has BOM{Input,Output}Stream classes which may help you here.

Another solution would be (ewwww) to always read/write using Windows-1252.

Other note: if you use Java 7, use the Files.newBuffered{Reader,Writer}() methods -- and the try-with-resources statement.