Checksum doesn't match when computed string contains accented characters

Question

First : I have a string which contains an accented character .

Second : I calcul the checksum for it .

 private static String checkSumInStream(String Str, String checksumAlgorithm) throws Exception
{           
    InputStream stream = new ByteArrayInputStream(Str.getBytes());
    MessageDigest digest = MessageDigest.getInstance(checksumAlgorithm);

    InputStream input = null;
    StringBuffer sb = new StringBuffer();
    try{
        input = stream;
        byte[] buffer = new byte[8192];
        do {
            int read = input.read(buffer);
            if(read <= 0)
                break;
            digest.update(buffer, 0, read);
        } while(true);
        byte[] sum = digest.digest();

        for (int i = 0; i < sum.length; i++) {
            sb.append(Integer.toString((sum[i] & 0xff) + 0x100, 16).substring(1));
        }

    }catch(IOException io)
    {

    }finally{
        if(input != null)
            input.close();
    }

    return sb.toString();
}

Then i write the string in text file and i I recalcul the checksum of the file

private String checkSum(File file,String checksumAlgorithm) throws Exception 
{
    MessageDigest digest = MessageDigest.getInstance(checksumAlgorithm);
    InputStream input = null;
    input = new FileInputStream(file);
    byte[] buffer = new byte[8192];
    do {
        int read = input.read(buffer);
        if(read <= 0)
            break;
        digest.update(buffer, 0, read);
    } while(true);
    input.close();
    byte[] sum = digest.digest();
    StringBuffer sb = new StringBuffer();
    for (int i = 0; i < sum.length; i++) {
        sb.append(Integer.toString((sum[i] & 0xff) + 0x100, 16).substring(1));
    }        
    return sb.toString();
}

--> Result : the comparison between checksum of an output steam and the file doesn't match when text contains an accented character .

Answer 1

How do you write the String to a file? You must be very careful to do that in the equivalent way of how you read it back from the file.

In your case:

OutputStream out = new FileOutputStream(myfile);
out.write(str.getBytes());
out.close();

Then it should work. But you need to keep in mind that str.getBytes() is not a safe method to use when you write to files, because it uses the platform default encoding for your characters. If you send such a file to some other place and use it there, you may be reading it back with the wrong encoding.

And it's possible that your platform default encoding doesn't even support accented characters! (But if you write and read files in exactly the same way, then you should get exactly the same result, so this wouldn't be the cause of your problem)

The best thing to do is to use the UTF-8 character encoding. Where ever you used str.getBytes(), replace it with str.getBytes("UTF-8"), or str.getBytes(Charset.forName("UTF-8")) if you want to avoid having to catch UnsupportedEncodingException [even though every Java implementation is required to support the UTF-8 encoding. It's annoying...]