Creating a multi-part MIME message with a binary component for an HTTP request in Python 3

Question

I'm trying to encode a binary file with MIMEApplication in Python 3.3, as part of a multi-part MIME HTTP POST. I have a problem that character 0x0d gets reinterpreted as a newline 0xa, despite everything being set to binary bytes.

Here's a minimal test scenario, with a binary string with a 0x0d in it, getting misinterpreted:

from email.encoders import encode_noop
from email.generator import BytesGenerator
from email.mime.application import MIMEApplication
import io

app = MIMEApplication(b'Q\x0dQ', _encoder=encode_noop)
b = io.BytesIO()
g = BytesGenerator(b)
g.flatten(app)
for i in b.getvalue()[-3:]:
    print("%02x " % i, end="")
print()

Output is: 51 0a 51 when it should be 51 0d 51

Note that this is to generate a binary part for a multipart http POST message.

Answer 1

I was able to solve my problem by putting a dummy 'marker' in as the MIMEApplication contents, then substituting in the real binary text after generating the MIME message:

from email.encoders import encode_noop
from email.generator import BytesGenerator
from email.mime.application import MIMEApplication
import io

# Actual binary "file" I want to encode (in real life, this is a file read from disk)
bytesToEncode = b'Q\x0dQ'

# unique marker that we can find and replace after message generation
binarymarker = b'GuadsfjfDaadtjhqadsfqerasdfiojBDSFGgg'

app = MIMEApplication(binarymarker, _encoder=encode_noop)
b = io.BytesIO()
g = BytesGenerator(b)
g.flatten(app, linesep='\r\n')  # linesep for HTTP-compliant header line endings

# replace the marker with the actual binary data, then you have the output you want!
body = b.getvalue().replace(binarymarker, bytesToEncode)

After this, body has the value I want, without messing up the binary bytes:

b'Content-Type: application/octet-stream\r\nMIME-Version: 1.0\r\n\r\nQ\rQ'

For a multi-part message, you just assemble the multipart message first, then do the replace() at the end.

Answer 2

Try following (not specifying encoder, using default base64 encoder):

import email
from email.encoders import encode_noop
from email.generator import BytesGenerator
from email.mime.application import MIMEApplication
import io

app = MIMEApplication(b'Q\x0dQ')
b = io.BytesIO()
g = BytesGenerator(b)
g.flatten(app)
msg = email.message_from_bytes(b.getvalue())
assert msg.get_payload(decode=True) == b'Q\x0dQ'