javax.crypto.BadPaddingException: data hash wrong

Question

I try to write a crypto package in java that wraps the javax.crypto package and makes it more comfortable to use. So I implemented a RSACore class that only wraps javax.crypto. Then I implemented two classes RSAEncoder and RSADecoder to hide tha javax.crypto.Cipher and initiation of it.

Now to my problem: I have written some small tests for RSACore, RSAEncoder and RSADecoder. All tests work fine except the decoder's one. I get everytime the BadPaddingException when I call the dofinal() on the encoded data. I googled for it and find in most cases it has to do with wrong keys but I am sure I only generate one pair and work on it. I do not know where to search the error. As provider I use bouncycastle because with JCE I got an error that signing is not possible with OAEP Here is the complete errormessage:

javax.crypto.BadPaddingException: data hash wrong
at org.bouncycastle.jcajce.provider.asymmetric.rsa.CipherSpi.engineDoFinal(Unknown Source)
at javax.crypto.Cipher.doFinal(Cipher.java:2145)
at de.coding.crypto.rsa.RSACore.doFinal(RSACore.java:178)
at de.coding.crypto.rsa.RSADecoder.doFinal(RSADecoder.java:59)
at de.coding.crypto.interfaces.AbstractDecoder.doFinal(AbstractDecoder.java:49)
at de.coding.crypto.rsa.RSADecoderTest.test(RSADecoderTest.java:21)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

here is the test:

package de.coding.crypto.rsa;

import static org.junit.Assert.*;

import java.security.KeyPair;

import org.junit.Test;

public class RSADecoderTest
{
    @Test
    public void test()
    {
        try
        {
            KeyPair kp = RSACore.generateKeys();
            String str = "Hello world! This is me. Life should be fun for everyone.";
            RSAEncoder encoder = new RSAEncoder(kp.getPublic());
            byte[] encoded = encoder.doFinal(str.getBytes());
            RSADecoder decoder = new RSADecoder(kp.getPrivate());
            byte[] decoded = decoder.doFinal(encoded);//<-------------------BadPaddingException
            assertTrue(str.getBytes().length == decoded.length);
            int size = decoded.length;
            for (int i=0; i<size; i++)
            {
                assertTrue(decoded[i] == str.getBytes()[i]);
            }
        }
        catch (Exception e)
        {
            fail();
        }
    }
}

Do you see there any logical error? I also can post encoder, decoder and core but that are plenty of lines. Thank you for reading and helping!

RSAEncoder:

public RSAEncoder(PublicKey key) throws InvalidKeyException
{
    super();
    cipher = RSACore.initEncodeMode(key);
}

public byte[] doFinal()
{
    try
    {
        buffer.add(RSACore.doFinal(cipher));//buffer is a LinkedList<byte[]>
        return clearBuffer();
    }
    catch (IllegalBlockSizeException e)
    {
        throw new RuntimeException(e);
    }
    catch (BadPaddingException e)
    {
        throw new RuntimeException(e);
    }
}

protected byte[] clearBuffer()
{
    byte[] ret = new byte[getBufferSize()];
    int index = 0;
    for (byte[] b : buffer)
    {
        if (b != null)
        {
            for (int i=0; i<b.length; i++)
            {
                ret[index] = b[i];
            }
        }
    }
    buffer.clear();
    return ret;
}

public byte[] doFinal(byte[] b, int offset, int length)
{
    if (length > RSACore.KEYSIZE-11)
    {
        update(b, offset, length);
        return doFinal();
    }
    else
    {
        try
        {
            buffer.add(RSACore.doFinal(cipher, b, offset, length));
            return clearBuffer();
        }
        catch (IllegalBlockSizeException e)
        {
            throw new RuntimeException(e);
        }
        catch (BadPaddingException e)
        {
            throw new RuntimeException(e);
        }
    }
}

RSADecoder (same except condtructor):

public RSADecoder(PrivateKey key) throws InvalidKeyException
{
    cipher = RSACore.initDecodeMode(key);
}

RSACore:

public byte[] doFinal(byte[] b)
{
    return doFinal(b, 0, b.length);
}

public static byte[] doFinal(Cipher cipher, byte[] msg, int offset, int length) throws IllegalBlockSizeException, BadPaddingException
{
    byte[] ret = cipher.doFinal(msg, offset, length);
    if (ret == null)
    {
        return new byte[0];
    }
    return ret;
}

public static Cipher initEncodeMode(Key key) throws InvalidKeyException
{
    try
    {
        Cipher cipher = Cipher.getInstance(ALGORITHMDETAIL, PROVIDER);
        cipher.init(Cipher.ENCRYPT_MODE, key);
        return cipher;
    }
    catch (NoSuchAlgorithmException e)
    {
        throw new RuntimeException(e);
    }
    catch (NoSuchPaddingException e)
    {
        throw new RuntimeException(e);
    }
    catch (NoSuchProviderException e)
    {
        throw new RuntimeException(e);
    }
}

public static Cipher initDecodeMode(Key key) throws InvalidKeyException
{
    try
    {
        Cipher cipher = Cipher.getInstance(ALGORITHMDETAIL, PROVIDER);
        cipher.init(Cipher.DECRYPT_MODE, key);
        return cipher;
    }
    catch (NoSuchAlgorithmException e)
    {
        throw new RuntimeException(e);
    }
    catch (NoSuchPaddingException e)
    {
        throw new RuntimeException(e);
    }
    catch (NoSuchProviderException e)
    {
        throw new RuntimeException(e);
    }
}

Answer 1

I'm not sure why you need a linked list, but the clearBuffer() method is obviously wrong:

protected byte[] clearBuffer() {
    byte[] ret = new byte[getBufferSize()];
    int index = 0;
    for (byte[] b : buffer)
    {
        if (b != null)
        {
            for (int i=0; i<b.length; i++)
            {
                ret[index] = b[i];
            }
        }
    }
    buffer.clear();
    return ret;
}

Except the first byte, all the other ones will be 0, since index never changes in this loop.