Why does verify smime signature fail?

Question

I've got an email with a signature which I'm trying to verify.

I am using openssl openssl smime -inform SMIME -CAfile all.pem -verify -in signed.eml to check, which gives errors:

4144:error:0D07209B:asn1 encoding routines:ASN1_get_object:too long:asn1_lib.c:142:
4144:error:0D068066:asn1 encoding routines:ASN1_CHECK_TLEN:bad object header:tasn_dec.c:1302:
4144:error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:379:Type=X509_SIG
4144:error:21071069:PKCS7 routines:PKCS7_signatureVerify:signature failure:pk7_doit.c:978:
4144:error:21075069:PKCS7 routines:PKCS7_verify:signature failure:pk7_smime.c:312:

After googling around for some time, this first seemed to be an error with the asn.1 structure. But running openssl asn1parse -i -in sig.txt (sig.txt being only the signature from signed.eml from before) gives no errors and shows the ASN.1 tree (IMHO) nicely.

And as a sidenote: the email is shown as successfully verified in MS Outlook 2010.

Any ideas what could be wrong with the signature or which pecularities of openssl I'm hitting here? Any other tools besides openssl to check the signature (to rule out an openssl problem)?

Answer 1

Ok, so with help from the openssl mailing list:

It is an error with the asn.1 structure. The signature was malformed. That wouldn't cause problems with asn1parse but would when OpenSSL tried to verify it.
It didn't have a proper DigestInfo structure but just the raw SHA1 hash, which violates the RSA scheme mentioned in the PKCS#1 specification.

Outlook (and other clients) seem to tolerate the raw form, whereas openssl (and Thunderbird and other clients) don't.