Verifying Authenticode signed executables and DLLs using OpenSSL API

Question

I have installed openssl and now the rsa_test.c is running fine. What I want to do is:

1. Open any exe or dll digital certificate. Extract the Thumbprint and PublicKey. The public key contains the exponent too (if you are familiar with ASN1 encoding).

2. Now I have already calculated SHA-1 hash of the file and compared it to the MD field to find it correct. So now I want to pass the Thumbprint, PublicKey->m and PublicKey->e to RSA_public_decrypt function to calculate the decrypted SHA-1 hash of the file.

What Padding Algorithm should I pass to the function? When I pass RSA_NO_PADDING, it decrypts but give me the 128-bytes long hash which is not at all correct. For all other padding algorithm options it returns the error of padding not found.

Would you please teach me or tell me the correct parameters to pass through the function with an example or good link to example?

Answer 1

You should use the PKCS7_verify function to verify the whole PKCS #7 signedData block.

You need two steps to completely verify the signature (see Microsoft specs):

1. Verify the integrity and identity of the PKCS #7 signature.
2. Calculate the Message Digest and compare it to the one specified in the signature.

Note that the digest is not limited to SHA-1 in the specs. The digest algorithm is contained in digestAlgorithm field in the signature.