iconv equivalent code in Java doesn't return same results

Question

I have requirement to encode a file from UTF-8 to Shift_JIS. Previously, this was done using the iconv command as below

iconv -f utf8 -t sjis $INPUT_FILE 

The input file I supply returns an error, saying

Illegal input sequence at position 2551

I have written this Java code:

FileInputStream fis = new FileInputStream(
        "Input.txt");
InputStreamReader in = new InputStreamReader(fis, "UTF-8");
FileOutputStream fos = new FileOutputStream("Output.txt");
OutputStreamWriter out = new OutputStreamWriter(fos, "Shift_JIS");

        int val = 0;
        StringBuilder sb = new StringBuilder();

        while((val =in.read() )!= -1){
            System.out.println(Integer.toHexString(val));
            sb.append((char)val);
        }
        out.write(sb.toString());
        out.flush();
        fis.close();
        out.close();

The code executes fine with the same input file and doesn't return any error.

Am I missing anything here?

Joachim. that looks like the answer. i have added my code in the question. I am now getting unmappable character error. But it fails to encode normal characters like any text "hello". am i doing it wrong anywhere

    private static CharsetDecoder decoder(String encoding) {
        return Charset.forName(encoding).newDecoder()
            .onMalformedInput(CodingErrorAction.REPORT)
            .onUnmappableCharacter(CodingErrorAction.REPORT);
    }

    private static CharsetEncoder encoder(String encoding) {
        return Charset.forName(encoding).newEncoder()
            .onMalformedInput(CodingErrorAction.REPORT)
            .onUnmappableCharacter(CodingErrorAction.REPORT);
    }

    public static void main(String[] args) throws IOException {
        FileInputStream fis = new FileInputStream(
        "D:\\Input.txt");
InputStreamReader in = new InputStreamReader(fis, decoder("UTF-8"));
FileOutputStream fos = new FileOutputStream("D:\\Output.txt");
OutputStreamWriter out = new OutputStreamWriter(fos, encoder("Shift_JIS"));
        char[] buffer = new char[4096];
        int length;

        while ((length = in.read(buffer)) != -1) {
            out.write(buffer, 0, length);
        }

        out.flush();
    }

Answer 1

That should be merely a problem concerning UTF-8. Just do an InputStream and start hex dumping from position 2551, or a bit earlier for preceding text.

Especially interesting is, what iconv delivers there.

---

A dump:

So we can see which data caused the problem.

public static void main(String[] args) {
    try (BufferedInputStream in = new BufferedInputStream(
            new FileInputStream("D:\\input.txt"))) {
        dumpBytes(in, 2551 - 10, 20);
    } catch (IOException ex) {
        ex.printStackTrace();
    }
}

private static void dumpBytes(InputStream in, long offset, int length)
        throws IOException {
    long pos = in.skip(offset);
    while (length >= 0) {
        int b = in.read();
        if (b == -1) {
            break;
        }
        b &= 0xFF;
        System.out.printf("%6d: 0x%02x %s '%c'%n", pos, b,
            toBinaryString(b), (32 <= b && b < 127 ? (char)b : '?'));

        --length;
        ++pos;
    }
}

private static String toBinaryString(int b) {
    String s = Integer.toBinaryString(b);
    s = "00000000" + s;
    s = s.substring(s.length() - 8);
    s = s.substring(0, 4) + "_" + s.substring(4);
    return s;
}