Unable to convert bytes from charset 65535 in to Japanese (5035)

Question

I have a character set conversion issue:

I am updating Japanese Kanji characters in DB2 in iSeries system with the following conversion method:

AS400 sys = new AS400("<host>","username","password");
CharConverter charConv = new CharConverter(5035, sys);
byte[] b = charConv.stringToByteArray(5035, sys, "試験");
AS400Text textConverter = new AS400Text(b.length, 65535,sys);

While retrieving, I use the following code to convert & display:

CharConverter charConv = new CharConverter(5035, sys);
byte[] bytes = charConv.stringToByteArray(5035, sys, dbRemarks);
String s = new String(bytes);
System.out.println("Remarks after conversion to AS400Text :"+s);

But, the system is displaying garbled characters while displaying. Can anybody help me to decode Japanese characters from binary storage?

Answer 1

Thanks a lot Jon and Buck Calabro !

With your clues, I have succeeded with the following approach:

String remarks = new String(resultSet.getBytes("remarks"),"SJIS");
byte[] byteData = remarks.getBytes("SJIS");
CharConverter charConv = new CharConverter(5035, sys);
String convertedStr = charConv.byteArrayToString(5035, sys, byteData);

I am able to convert from string. I am planning to implement the same with JPA, and started coding.

Answer 2

Well I don't know anything about CharConverter or AS400Text, but code like this is almost always a mistake:

String s = new String(bytes);

That uses the platform default encoding to convert the binary data to text.

Usually storage and retrieval should go through opposite processes - so while you've started with a string and then converted it to bytes, and converted that to an AS400Text object when storing it, I'd expect you to start with an AS400Text object, convert that to a byte array, and then convert that to a String using CharConverter when fetching. The fact that you're calling stringToByteArray in both cases suggests there's something amiss.

(It would also help if you'd tell us what dbRemarks is, and how you've fetched it.)

I do note that having checked some documentation for AS400Text, I've seen this:

Due to recent changes in the behavior of the character conversion routines, this system object is no longer necessary, except when the AS400Text object is to be passed as a parameter on a Toolbox Proxy connection.

There's similar documentation for CharConverter. Are you sure you actually need to go through this at all? Have you tried just storing the string directly and retrieving it directly, without going through intermediate steps?

Answer 3

Thank you Jon Skeet!

Yes. I have committed a mistake, not encoding the string while declaration.

My issue is to get the data stored in DB2, convert it into Japanese and provide for editing in web page. I am getting dbRemarks from the result set. I have missed another thing in my post:

While inserting, I am converting to text like:

String text = (String) textConverter.toObject(b);
PreparedStatement prepareStatementUpdate = connection.prepareStatement(updateSql);
prepareStatementUpdate.setString(1, text);
int count = prepareStatementUpdate.executeUpdate();

I am able to retrieve and display clearly with this code:

String selectSQL = "SELECT remarks FROM empTable WHERE emp_id = ? AND dep_id=? AND join_date='2013-11-15' ";
prepareStatement = connection.prepareStatement(selectSQL);
prepareStatement.setInt(1, 1);
prepareStatement.setString(2, 1);
ResultSet resultSet = prepareStatement.executeQuery();
while ( resultSet.next() ) {
    byte[] bytedata = resultSet.getBytes( "remarks" );
    AS400Text textConverter2 = new AS400Text(bytedata.length, 5035,sys);
    String javaText = (String) textConverter2.toObject(bytedata);
    System.out.println("Remarks after conversion to AS400Text :"+javaText);
}

It is working fine with JDBC, but for working with JPA, I need to convert to string for editing in web page or store in table. So, I have tried this way, but could not succeed:

String remarks = resultSet.getString( "remarks" );
byte[] bytedata = remarks.getBytes();
AS400Text textConverter2 = new AS400Text(bytedata.length, 5035,sys);
String javaText = (String) textConverter2.toObject(bytedata);
System.out.println("Remarks after conversion to AS400Text :"+javaText);