How OpenSSL/Mozilla NSS RSA sign and verify (PK11_Sign, PK11_Verify, PK11_VerifyRecover functions) work?

Question

I have my RSA public and private keys (all of p,q,e,n,d) in PEM format. I am curious to know:

• How PK11_Sign(), PK11_Verify() and PK11_VerifyRecover() (from OpenSSL/Mozilla NSS library) work with RSA?
• How the padding is applied to the input message to be signed?

The context of my question is: I have seen PK11_Sign() adds some padding to my input data during signing. For example (given the key size is 162 bits):

my input = 31323334353036373839
padded input = 1FFFFFFFFFFFFFFFF0031323334353036373839

I would like to know:

• What is the name of this padding scheme and pointers on how it works?
• What is the default padding scheme for the above mentioned OpenSSL functions? For example, if I perform " openssl rsautl -in input.txt -inkey mykey.pem -out signed.txt ", which padding scheme will be used?

Answer 1

• PK11_Sign etc. uses PKCS#1 v.1.5 signatures, which includes the padding you mention.

• The padding scheme is part of the algorithm called EMSA-PKCS1-V1_5-ENCODE. I do not believe it has a name, although it might be informally called "PKCS#1 v.1.5 signature padding". It is defined in the PKCS#1 standard.

• According to the documentation the default for openssl rsautl is to use PKCS#1 v.1.5 signature, which implies this padding.