Streaming POST large request in Pylons

Question

I am working on a Pylons 1.0 project using apache with mod_wsgi. Ineed it to work with large POST and GET requests. For the GET requests I can just take the data source (file on disk usually) and read it and yield to the Pylons layer to stream the data back to the user. I also know that I can use mmap with urllib2 to stream data requests to other services.

However for POST requests to my service how do I stream the request to disk so as to not overwhelm my memory usage when someone uploads a large file? I see req.body_file which may indicate Pylons is already doing this for me. Anyone know if that is the case?

Answer 1

Pylons uses the WebOb project to supply the request and response objects, and it handles file uploads efficiently.

File upload handling internally is actually delegated to the python stdlib cgi module, which uses temporary files to handle uploads.

Ultimately, this is also due to the underlying WSGI standard, which specifies that the request input is a stream.