M2Crypto: extract message from opaque signed S/MIME (pkcs7-mime)

Question

I have a problem with extracting message data from opaque signed S/MIME message, like:

To: ngps@post1.com
From: ngps@mpost1.com
Subject: testing
MIME-Version: 1.0
Content-Disposition: attachment; filename="smime.p7m"
Content-Type: application/pkcs7-mime; smime-type=signed-data; name="smime.p7m"
Content-Transfer-Encoding: base64

MIIHQwYJKoZIhvcNAQcCoIIHNDCCBzACAQExCzAJBgUrDgMCGgUAMIICQwYJKoZI
hvcNAQcBoIICNASCAjANClMvTUlNRSAtIFNlY3VyZSBNdWx0aXB1cnBvc2UgSW50
ZXJuZXQgTWFpbCBFeHRlbnNpb25zIFtSRkMgMjMxMSwgUkZDIDIzMTJdIC0gDQpw
cm92aWRlcyBhIGNvbnNpc3RlbnQgd2F5IHRvIHNlbmQgYW5kIHJlY2VpdmUgc2Vj
dXJlIE1JTUUgZGF0YS4gQmFzZWQgb24gdGhlDQpwb3B1bGFyIEludGVybmV0IE1J
TUUgc3RhbmRhcmQsIFMvTUlNRSBwcm92aWRlcyB0aGUgZm9sbG93aW5nIGNyeXB0
b2dyYXBoaWMNCnNlY3VyaXR5IHNlcnZpY2VzIGZvciBlbGVjdHJvbmljIG1lc3Nh
Z2luZyBhcHBsaWNhdGlvbnMgLSBhdXRoZW50aWNhdGlvbiwNCm1lc3NhZ2UgaW50
ZWdyaXR5IGFuZCBub24tcmVwdWRpYXRpb24gb2Ygb3JpZ2luICh1c2luZyBkaWdp
dGFsIHNpZ25hdHVyZXMpDQphbmQgcHJpdmFjeSBhbmQgZGF0YSBzZWN1cml0eSAo
dXNpbmcgZW5jcnlwdGlvbikuDQoNClMvTUlNRSBpcyBidWlsdCBvbiB0aGUgUEtD
UyAjNyBzdGFuZGFyZC4gW1BLQ1M3XQ0KDQpTL01JTUUgaXMgaW1wbGVtZW50ZWQg
aW4gTmV0c2NhcGUgTWVzc2VuZ2VyIGFuZCBNaWNyb3NvZnQgT3V0bG9vay4NCqCC
AxAwggMMMIICdaADAgECAgECMA0GCSqGSIb3DQEBBAUAMHsxCzAJBgNVBAYTAlNH
MREwDwYDVQQKEwhNMkNyeXB0bzEUMBIGA1UECxMLTTJDcnlwdG8gQ0ExJDAiBgNV
BAMTG00yQ3J5cHRvIENlcnRpZmljYXRlIE1hc3RlcjEdMBsGCSqGSIb3DQEJARYO
bmdwc0Bwb3N0MS5jb20wHhcNMDAwOTEwMDk1ODIwWhcNMDIwOTEwMDk1ODIwWjBZ
MQswCQYDVQQGEwJTRzERMA8GA1UEChMITTJDcnlwdG8xGDAWBgNVBAMTD00yQ3J5
cHRvIENsaWVudDEdMBsGCSqGSIb3DQEJARYObmdwc0Bwb3N0MS5jb20wXDANBgkq
hkiG9w0BAQEFAANLADBIAkEAoz3zUF0dmxSU+1fso+eTdmjDY71gWNeXWX28qsBJ
0UFmq4JCtw7Gv4fJ0TZgQHVIrXgKrUvzsquu8eiVjuP/NwIDAQABo4IBBDCCAQAw
CQYDVR0TBAIwADAsBglghkgBhvhCAQ0EHxYdT3BlblNTTCBHZW5lcmF0ZWQgQ2Vy
dGlmaWNhdGUwHQYDVR0OBBYEFMcQhEeJ1x9d+8Rzag9yjCiutYKOMIGlBgNVHSME
gZ0wgZqAFPuHI2nrnDqTFeXFvylRT/7tKDgBoX+kfTB7MQswCQYDVQQGEwJTRzER
MA8GA1UEChMITTJDcnlwdG8xFDASBgNVBAsTC00yQ3J5cHRvIENBMSQwIgYDVQQD
ExtNMkNyeXB0byBDZXJ0aWZpY2F0ZSBNYXN0ZXIxHTAbBgkqhkiG9w0BCQEWDm5n
cHNAcG9zdDEuY29tggEAMA0GCSqGSIb3DQEBBAUAA4GBAKy2cWa2BF6cbBPE4ici
//wOqkLDbsI3YZyUSj7ZPnefghx9EwRJfLB3/sXEf78OHL7yV6IMrvEVEAJCYs+3
w/lspCMJC0hOomxnt0vjyCCd0JeaEwihQGbOo9V0reXzrUy8yNkwo1w8mMSbIvqh
+D5uTB0jKL/ml1EVLw3NJf68MYIBwTCCAb0CAQEwgYAwezELMAkGA1UEBhMCU0cx
ETAPBgNVBAoTCE0yQ3J5cHRvMRQwEgYDVQQLEwtNMkNyeXB0byBDQTEkMCIGA1UE
AxMbTTJDcnlwdG8gQ2VydGlmaWNhdGUgTWFzdGVyMR0wGwYJKoZIhvcNAQkBFg5u
Z3BzQHBvc3QxLmNvbQIBAjAJBgUrDgMCGgUAoIHYMBgGCSqGSIb3DQEJAzELBgkq
hkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTEzMDcxNTE1MDkzN1owIwYJKoZIhvcN
AQkEMRYEFI/KcwJXhIg0bRzYLfAtDhxRMzghMHkGCSqGSIb3DQEJDzFsMGowCwYJ
YIZIAWUDBAEqMAsGCWCGSAFlAwQBFjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcw
DgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3
DQMCAgEoMA0GCSqGSIb3DQEBAQUABEAPrw12PT+5T9kftixVi+MzE/SLZnOJqw9x
2q1YcztjY4drOJi6CH8bHOlfb1nXr/mBQuijLA5wFl22PDGTrlZ5

which is actually an opaque S/MIME signed (not encrypted) message opaque.p7 generated by M2Crypto's (0.21.1) sign function from demo/smime/test.py. The data obviously contains the message, it can be revealed e.g. by: openssl smime -verify -noverify -in opaque.p7.

Unfortunately, when I want to get the data with:

p7, data = M2Crypto.SMIME.smime_load_pkcs7('opaque.p7')

unfortunately, data is None. It's only a problem with the opaque S/MIME variant, because the same works for the clear.p7.

I guess this might be some compatibility problem, my OpenSSL version is 1.0.1e (Debian Wheezy).. I wonder if someone got it working.

UPDATE

Here is the output from modified test.py (original here), to demonstrate that both for clear and opaque S/MIME messages, M2Crypto correctly extracts the signer certificate, but extracts no data from the opaque SMIME:

test encrypt/decrypt... ok
test sign & save... ok
test load & verify opaque... ok
  DATA: ''
  SIGNERS: ['C=AU, ST=Some-State, O=Internet Widgits Pty Ltd']
test load & verify clear... ok
  DATA: 'actual message'
  SIGNERS: ['C=AU, ST=Some-State, O=Internet Widgits Pty Ltd']
test sign/verify... ok

test.py is patched, because it was failing on many places..

--- M2Crypto-0.21.1.orig/demo/smime/test.py 2011-01-15 20:10:06.000000000 +0100
+++ M2Crypto-0.21.1/demo/smime/test.py  2013-07-16 16:37:57.224845942 +0200
@@ -6,18 +6,7 @@
 
 from M2Crypto import BIO, Rand, SMIME, X509
 
-ptxt = """
-S/MIME - Secure Multipurpose Internet Mail Extensions [RFC 2311, RFC 2312] - 
-provides a consistent way to send and receive secure MIME data. Based on the
-popular Internet MIME standard, S/MIME provides the following cryptographic
-security services for electronic messaging applications - authentication,
-message integrity and non-repudiation of origin (using digital signatures)
-and privacy and data security (using encryption).
-
-S/MIME is built on the PKCS #7 standard. [PKCS7]
-
-S/MIME is implemented in Netscape Messenger and Microsoft Outlook.
-"""
+ptxt = 'actual message'
 
 def makebuf():
     buf = BIO.MemoryBuffer(ptxt)
@@ -27,14 +16,14 @@
     print 'test sign & save...',
     buf = makebuf()
     s = SMIME.SMIME()
-    s.load_key('client.pem')
-    p7 = s.sign(buf)
+    s.load_key('client_.pem')
+    p7 = s.sign(buf, flags=SMIME.PKCS7_DETACHED)
     out = BIO.openfile('clear.p7', 'w')
     out.write('To: ngps@post1.com\n')
     out.write('From: ngps@post1.com\n')
     out.write('Subject: testing\n')
     buf = makebuf() # Recreate buf, because sign() has consumed it.
-    s.write(out, p7, buf)
+    s.write(out, p7, buf, flags=SMIME.PKCS7_DETACHED)
     out.close()
 
     buf = makebuf()
@@ -50,36 +39,42 @@
 def verify_clear():
     print 'test load & verify clear...',
     s = SMIME.SMIME()
-    x509 = X509.load_cert('client.pem')
+    x509 = X509.load_cert('client_.pem')
     sk = X509.X509_Stack()
     sk.push(x509)
     s.set_x509_stack(sk)
     st = X509.X509_Store()
-    st.load_info('ca.pem')
+    st.load_info('client_.pem')
     s.set_x509_store(st)
     p7, data = SMIME.smime_load_pkcs7('clear.p7')
-    v = s.verify(p7)
-    if v:
+    data_s = data.read() if isinstance(data, BIO.BIO) else ''
+    v = s.verify(p7, BIO.MemoryBuffer(data_s))
+    if v and (v == ptxt):
         print 'ok'
     else:
         print 'not ok'
+    print '  DATA: %r' % (data_s,)
+    print '  SIGNERS: %r' % ([ x.get_subject().as_text() for x in p7.get0_signers(sk)],)
     
 def verify_opaque():
     print 'test load & verify opaque...',
     s = SMIME.SMIME()
-    x509 = X509.load_cert('client.pem')
+    x509 = X509.load_cert('client_.pem')
     sk = X509.X509_Stack()
     sk.push(x509)
     s.set_x509_stack(sk)
     st = X509.X509_Store()
-    st.load_info('ca.pem')
+    st.load_info('client_.pem')
     s.set_x509_store(st)
     p7, data = SMIME.smime_load_pkcs7('opaque.p7')
-    v = s.verify(p7, data)
-    if v:
+    data_s = data.read() if isinstance(data, BIO.BIO) else ''
+    v = s.verify(p7, makebuf()) # here we are verify against ptxt, since we get no data
+    if v and (v == ptxt):
         print 'ok'
     else:
         print 'not ok'
+    print '  DATA: %r' % (data_s,)
+    print '  SIGNERS: %r' % ([ x.get_subject().as_text() for x in p7.get0_signers(sk)],)
     
 def verify_netscape():
     print 'test load & verify netscape messager output...',
@@ -102,31 +97,32 @@
     s = SMIME.SMIME()
 
     # Load a private key.
-    s.load_key('client.pem')
+    s.load_key('client_.pem')
 
     # Sign.
-    p7 = s.sign(buf)
+    p7 = s.sign(buf, flags=SMIME.PKCS7_DETACHED)
 
     # Output the stuff.
+    buf = makebuf()
     bio = BIO.MemoryBuffer()
-    s.write(bio, p7, buf)
+    s.write(bio, p7, buf, flags=SMIME.PKCS7_DETACHED)
     
     # Plumbing for verification: CA's cert.
     st = X509.X509_Store()
-    st.load_info('ca.pem')
+    st.load_info('client_.pem')
     s.set_x509_store(st)
 
     # Plumbing for verification: Signer's cert.
-    x509 = X509.load_cert('client.pem')
+    x509 = X509.load_cert('client_.pem')
     sk = X509.X509_Stack()
     sk.push(x509)
     s.set_x509_stack(sk)
 
     # Verify.
     p7, buf = SMIME.smime_load_pkcs7_bio(bio)
-    v = s.verify(p7, flags=SMIME.PKCS7_DETACHED)
-    
-    if v:
+    v = s.verify(p7, buf, flags=SMIME.PKCS7_DETACHED)
+
+    if v and (v == ptxt):
         print 'ok'
     else:
         print 'not ok'

client.pem contains expired certificate, so it is using self-signed client_.pem, which was generated by openssl req -new -x509 -newkey rsa -nodes -keyout client_.pem -out client_.pem

(Please note that test load & verify opaque testcase pretends to be succeeding, since the verify step is changed to be verifying against the original known text)

Answer 1

What I think is the main misunderstanding here is that in the case of MIME types other than "multipart/signed", smime_load_pkcs7() is meant to return None in the second part of the tuple, and SMIME.SMIME.verify() is meant to take None for its second parameter.

When you call s.verify(a, b), it doesn't mean "verify that the signature in a matches the message b", it means "verify that the PKCS7 structure a has a valid signature, given the x509 store and stack on the SMIME object s. Oh, and if a has a detached message, you can find the message part in b."

If you'd like to fix your verify_opaque() function, take out the data_s stuff altogether, and put back the original s.verify(p7, data) call. If v == ptxt, then the extraction of the original message went as expected and the signature is verified. If you still want to print out the "DATA:" line, use v instead of data_s.

If your question is about how to get the original message data from an arbitrary opaque signed message, it looks like you are expected to get a valid M2Crypto.SMIME.SMIME context s and call s.verify() with it! This is not an operation I've needed to do myself before, so I might be missing some simple API, but as far as I can tell M2Crypto doesn't expose any simpler way. At least, if you don't have a valid set of certs to check against, or you don't care, you can pass the PKCS7_NOVERIFY flag to the s.verify() call just as you did with the openssl cmdline tool.