Basecamp file upload using Python

Question

According to the Basecamp API documentation, files should be uploaded using HTTP POST with content type set to application/octet-stream and the request body containing the binary contents of the file (see http://developer.37signals.com/basecamp/). I'd like to stream the file rather than reading the whole thing into memory. I'm using Python 2.7.

I can see a few possibilities:

• Do this using the low-level socket API.
• Use urllib2 with Poster (http://atlee.ca/software/poster/) to handle the file streaming. However, with Poster you register special openers for the file streaming and I'm already using my own opener explicitly (passing it to build_opener) to handle authentication on the Basecamp server. Also Poster docs only talk about posting form data and it's not clear to me yet (still reading the source code) whether it can handle octet-stream.
• Use httplib. This looks like it will give me more low-level handling for the POST data (so I can use octet-stream) but I still don't see an easy way to stream the file.

I found Python: HTTP Post a large file with streaming but it sounds like unless I want to use a form data format I'd have to patch httplib (!). That post is a year old though so I'm hoping that there is now a better way.

Right now I'm investigating creating my own mixin like Poster does, but wondering: is this really so hard? Isn't there an easier way to handle what seems to me like a relatively standard use case?

Answer 1

I ended up using Twisted for this since I needed the upload to happen asynchronously anyway. This excellent blog post explains the general procedure: http://marianoiglesias.com.ar/python/file-uploading-with-multi-part-encoding-using-twisted/. I simply wrote my own producer instead to write raw binary data as the POST payload.