Java convert Windows-1252 to UTF-8, some letters are wrong

Question

I receive data from a external Microsoft SQL 2008 database (I make queries with MyBatis). The data is encoded as "Windows-1252".

I have tried to re-encode to UTF-8:

String textoFormado = ...value from MyBatis... ; 
String s = new String(textoFormado.getBytes("Windows-1252"), "UTF-8");

Almost the whole string is correctly decoded, but some letters with accents are not.

For example:

1. I received this: �vila
2. The code above makes: �?vila
3. I expected: Ávila

Answer 1

I solved it thanks to all.

I have the next project structure:

• MyBatisQueries: I have a query with a "select" which gives me the String
• Pojo to save the String (which gave me the String with conversion problems)
• The class which uses the query and the Pojo object with data (that showed me bad decoded)

at first I had (MyBatis and Spring inject dependencies and params):

public class Pojo {
    private String params;
    public void setParams(String params) {
        try {
            this.params = params;
        }
    }

}

The solution:

public class Pojo {
    private String params;
    public void setParams(byte[] params) {
        try {
            this.params = new String(params, "UTF-8");
        } catch (UnsupportedEncodingException e) {
            this.params = null;
        }
    }

}

Answer 2

Obviously, textoFormado is a variable of type String. This means that the bytes were already decoded. Java then internally uses a 16-bit Unicode representation. What you did, is to encode your string with Windows-1252 followed by reading the resulting bytes with an UTF-8 encoding. That does not work.

What you need is the correct encoding when reading the bytes:

byte[] sourceBytes = getRawBytes();
String data = new String(sourceBytes , "Windows-1252");

For using this string inside your program, you do not need to do anything. Simply use it. If - however - you want to write the data back to a file for example, you need to encode again:

byte[] destinationBytes = data.getBytes("UTF-8");
// write bytes to destination file here

Answer 3

Why not tackling the issue at a lower level: reading the String in proper encoding from your database.

Most JDBC connection-string or URIs support the property characterEncoding.

So in you Microsoft SQL Server case you could have for example jdbc:sqlserver://localhost:52865;databaseName=myDb?characterEncoding=utf8.

Then each String column should be read in the specified encoding without the need to (re-)convert it manually to it.

See also:

• JDBC character encoding
• Problems reading/writing UTF-8 data in MySQL from Java using JDBC connector 5.1