Error when decrypting with OpenSSL rsa key

Question

I am trying to write a simple application to encrypt/decrypt a buffer of unsigned char using OpenSSL RSA encryption. I have my public key and encrypt an array with

unsigned char plain [13] = "Hello World!";
unsigned char encrypted[1024]={};
unsigned char decrypted[1024]={};
int padding = RSA_PKCS1_OAEP_PADDING;
int flen = 13;
int res = RSA_public_encrypt(flen, plain, encrypted, rsa_pbk, padding);

where rsa_pbk is an RSA structure that contains the key. If I print it, the result is the following hexadecimal array:

13 d0 44 a3 2b 12 67 d8 e2 aa cf 53 6c 81 ed e9 9e 2d 9c dd 1d 28 84 5b 60 93 58 1c 7f eb b 66 26 39 8c 27 48 11 31 6 53 90 16 2e da 5c 7e 48 3e 15 c2 19 d3 10 79 71 1a fa f7 c1 57 93 82 f2 95 1 e d8 70 ba 1b 7e 12 d5 a 34 75 8f 2f 3c a6 60 f1 4b 60 6c 94 3e 4b 72 61 81 fb 89 e2 1e 5a 8 48 55 a5 5f 44 3b a4 e2 16 eb 7e 87 10 18 2e 1b 82 e7 86 43 69 21 ec a5 98 4 de 90 c5 5a 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

I am a little bit suspicious of the zeros at the end, but it could be ok. But, when I try to decrypt it with

flen = keysize - 50;
RSA_private_decrypt(flen, encrypted, decrypted, this->rsa_pvk, padding);

I get the following error

error:0407A079:rsa routines:RSA_padding_check_PKCS1_OAEP:oaep decoding error
error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed

that I'm not able to understand. Does anybody know what it means and why it arises?

Answer 1

Manual page states that: RSA_public_encrypt() returns the size of the encrypted data. RSA_private_decrypt() returns the size of the recovered plaintext. On error, -1 is returned; the error codes can be obtained by ERR_get_error(3).

Your code should look like this:

unsigned char plain [13] = "Hello World!";
unsigned char encrypted[1024]={};
unsigned char decrypted[1024]={};
int padding = RSA_PKCS1_OAEP_PADDING;
int flen = 13;
int res = RSA_public_encrypt(flen, plain, encrypted, rsa_pbk, padding);
flen = res;
res = RSA_private_decrypt(flen, encrypted, decrypted, this->rsa_pvk, padding);