How to add per-file custom header to multipart upload with requests?

Question

python-requests appears to support custom collections as a list/mapping of files to upload.

This in turn should allow me to add custom header fields with each file being uploading within one multipart upload.

Nope. Each file in the list/mapping must be a file object (or contents string), a 2-tuple of a filename and file object (or contents), or a 3-tuple of a filename, file object (or contents) and file type. Anything else is illegal.

Unless #1640 gets accepted upstream, in which case all you'll have to do is use a 4-tuple of a filename, file object (or contents), file type, and header dictionary.

The right thing to do at this point is probably to use a different library. For example, if you use urllib3 directly instead of through the request wrappers, it makes what you want to do reasonably easy. You just have to deal with all the extra verbosity of using urllib3 instead of requests.

And meanwhile, you can file a feature request against requests and an easier way may get added in the future.

But it's a bit frustrating that the functionality you need is all there under the covers, you just can't get to it.

It looks like doing things cleanly would be a nightmare. But it's all pretty simple code, and we can hack our way down to it pretty easily, so let's do that.

Take a look at requests.PreparedRequest.prepare_body. It expects each file in files to be a tuple of filename, contents or file object, and optionally content-type. It basically just reads in any file objects to convert them to contents, and passes everything straight through to urllib3.filepost.encode_multipart_formdata. So, unless we want to replace this code, we're going to need to smuggle the headers in with one of these values. Let's do that by passing (filename, contents, (content_type, headers_dict)). So, requests itself is unchanged.

What about that urllib3.filepost.encode_multipart_formdata that it calls? As you can see, if you pass it tuples for the files, it calls a function called iter_field_objects, which ultimately calls urllib3.fields.RequestField.from_tuples on each one. But if you look at the from_tuples alternate constructor, it says it's there to handle an "old-style" way of constructing RequestField objects, and the normal constructor is there for a "new-style" way, which actually does let you pass in headers.

So, all we need to do is monkeypatch iter_field_objects, replacing its last line with one that uses the new-style way, and we should be done. Let's try:

import requests
import requests.packages.urllib3
from requests.packages.urllib3.fields import RequestField, guess_content_type
import six

old_iter_field_objects = requests.packages.urllib3.filepost.iter_field_objects
def iter_field_objects(fields):
    if isinstance(fields, dict):
        i = six.iteritems(fields)
    else:
        i = iter(fields)

    for field in i:
      if isinstance(field, RequestField):
        yield field
      else:
        name, value = field
        filename = value[0]
        data = value[1]
        content_type = value[2] if len(value)>2 else guess_content_type(filename)
        headers = None
        if isinstance(content_type, (tuple, list)):
            content_type, headers = content_type
        rf = RequestField(name, data, filename, headers)
        rf.make_multipart(content_type=content_type)
        yield rf
requests.packages.urllib3.filepost.iter_field_objects = iter_field_objects

And now:

>>> files = {'file': ('foo.txt', 'foo\ncontents\n'), 
...          'file2': ('bar.txt', 'bar contents', 'text/plain'),
...          'file3': ('baz.txt', 'baz contents', ('text/plain', {'header': 'value'}))}
>>. r = request.Request('POST', 'http://example.com', files=files)
>>> print r.prepare().body
--1ee28922d26146e7a2ee201e5bf22c44
Content-Disposition: form-data; name="file3"; filename="baz.txt"
Content-Type: text/plain
header: value

baz contents
--1ee28922d26146e7a2ee201e5bf22c44
Content-Disposition: form-data; name="file2"; filename="bar.txt"
Content-Type: text/plain

bar contents
--1ee28922d26146e7a2ee201e5bf22c44
Content-Disposition: form-data; name="file"; filename="foo.txt"
Content-Type: text/plain

foo

Tada!

Note that you need relatively up-to-date requests/urllib3 for this to work. I think requests 2.0.0 is sufficient.