to use RSA_PKCS1_OAEP_PADDING for RSA signature

Question

Previously I was using rsa signature with no padding ,but now I was asked to add PKCS1_OAEP padding ,Initially I tried simple adding this flag "RSA_PKCS1_OAEP_PADDING", but it is giving error code while running like this

error:04066076:rsa routines:RSA_EAY_PRIVATE_ENCRYPT:unknown padding type

Further I googled about the rsa signature with PKCS1_OAEP padding but didn't get anything except it is said that for implementing RSA_PKCS1_OAEP padding you have to use this one

int RSA_padding_add_PKCS1_OAEP(unsigned char *to, int tlen,
           unsigned char *f, int fl, unsigned char *p, int pl);

and then do private_encrypt with RSA_NO_padding but still having confusion as they have not explained clearly how to use this padding_add function explictly .can anyone provide help .would really appreciate that .

thanks in advance

Answer 1

According to RFC 3447 OAEP padding scheme can be used only with encryption operation and therefore it cannot be used with signatures. If standard RSASSA-PKCS1-v1_5 scheme is not good enough for you I believe you should use RSASSA-PSS which is recommended for new applications and have characteristics similar to OAEP scheme. See RFC 3447 for more details.

You can easily check out that my answer is correct with command line OpenSSL tool:

1. Generate private key:

   openssl genrsa -out private.key 2048
   

2. Generate some input data:

   echo "Hello world" > input.data
   

3. Try to generate signature with OAEP scheme:

   openssl rsautl -sign -oaep -inkey private.key -in input.data -out output.data
   RSA operation error
   139655304349344:error:04066076:rsa routines:RSA_EAY_PRIVATE_ENCRYPT:unknown padding type:rsa_eay.c:389:
   

4. Try to encrypt data with OAEP scheme:

   openssl rsautl -encrypt -oaep -inkey private.key -in input.data -out output.data
   

You can also take a look at "rsautl" module source code if you need more information but remember that RFC 3347 is your friend :)