cryptoapi and openssl

Question

I'm trying to encrypt and sign a file with cryptoapi with some X.509 certificates. I want to verify and decrypt this file with openssl.

On windows I think I need to use the CryptSignAndEncryptMessage function to encrypt and sign data. I used this example from MSDN to create a signed and encrypted message.

How can I decrypt/verify this file using openssl? I removed the first 4 bytes from the message since it contained the length of the message (from the windows blob). When I call openssl -asn1parse I get some output that indicates it to be parsable by openssl.

When trying to verify the signature with openssl I recieve an error:

openssl rsautl -verify -inkey AlonsoCert.pem -keyform pem -certin -in sandvout-without-4byte.txt
RSA operation error
3073579208:error:0406706C:rsa routines:RSA_EAY_PUBLIC_DECRYPT:data greater than mod len:rsa_eay.c:680:

Answer 1

CryptSignAndEncrypt message seems to use RC4 cipher with empty ASN.1 parameters field and, looking at OpenSSL sources, openssl chokes on try to generate IV (which is not needed for RC4).

Try to use other cipher (AES for example) in CryptAndSignMessage. Anyway, RC4 is very old, insecure, and obsolete.

Answer 2

Your ASN.1 dump information shows you've created a PKCS#7 CMS output from your CryptoAPI code. As a result you cannot use the basic OpenSSL decryption and verification methods.

Instead, use the cms mode:

openssl cms -decrypt -inform DER -in sandvout-without-4byte.txt 
    -out decrypted.bin -recip testkey.pfx

(Note: I've not used this mode before, so I think the syntax I've suggested is correct. Either way, this should hopefully be the step in the right direction that solves this.)

Answer 3

Try using openssl smime to verify and/or decrypt. The syntax is fairly straight-forward but you can find the information here: http://www.openssl.org/docs/apps/smime.html