Convert ANSI characters to UTF-8 in Java

Question

Is there a way to convert an ANSI string to UTF using Java.

I have a custom serializer that uses readUTF & writeUTF methods of the DataInputStream class to deserialize and serialze string. If i receive a string encoded in ANSI and is too long, ~100000 chars long i get the error;

Caused by: java.io.UTFDataFormatException: encoded string too long: 106958 bytes

However in my Junit tests i'm able create a string with 120000 'a's and it works perfectly

I have checked the following posts but still having errors;

• Converting UTF-8 to ISO-8859-1 in Java - how to keep it as single byte
• How do I replace accented Latin characters in Ruby?

Answer 1

This error is not caused by character encoding. It means the length of the UTF data is wrong.

EDIT: Just realized this is a writing error, not reading error.

The UTF length is only 2 bytes so it can only hold 64K UTF-8 bytes. You are trying to writing 100K, it's not going to work.

This limit is hardcoded and no way to get around this,

if (utflen > 65535)
    throw new UTFDataFormatException(
            "encoded string too long: " + utflen + " bytes");

Answer 2

byte[] asciiBytes = ...;
String unicode = new String(asciiBytes, "US-ASCII");
byte[] utfBytes = unicode.getBytes("UTF-8");

Answer 3

Which ANSI codepage? There are lots of different character encodings which all refer to "ANSI". The DOS codepage is 437 (without the drawing symbols). If you use codepage 850, this will work:

String unicode = new String(bytes, "IBM850");

(where bytes is an array with the ANSI characters). After that, you can convert this string into a byte array with any encoding using unicode.getBytes(encoding).

Windows often uses the codepage 1252 (use "windows-1252" for that).

Answer 4

ZZ Coder already answered the question, but I have written a more detailed explanation and suggesting a workaround on this blog. Basically, the problem is in DataOutputStream, because it restricts the writeable String to 64KB. There are other possible workarounds to bystep the issue, some might work without breaking the actual binary data format one is using...