Twisted web - Keep request data after responding to client

Question

I have a front-end web server written in Twisted Web, that interfaces with another web server. Clients upload files to my front-end server, which then sends the files along to the back-end server. I'd like to receive the uploaded file, and then send an immediate response to the client before sending the file on to the back-end server. That way the client doesn't have to wait for both uploads to occur before getting a response.

I'm trying to do this by starting the upload to the back-end server in a separate thread. The problem is, after sending a response to the client, I'm no longer able to access the uploaded file from the Request object. Here's an example:

class PubDir(Resource):

    def render_POST(self, request):
        if request.args["t"][0] == 'upload':
            thread.start_new_thread(self.upload, (request,))

        ### Send response to client while the file gets uploaded to the back-end server:
        return redirectTo('http://example.com/uploadpage')

    def upload(self, request):
        postheaders = request.getAllHeaders()
        try:
            postfile = cgi.FieldStorage(
                fp = request.content,
                headers = postheaders,
                environ = {'REQUEST_METHOD':'POST',
                         'CONTENT_TYPE': postheaders['content-type'],
                        }
                )
        except Exception as e:
            print 'something went wrong: ' + str(e)

        filename = postfile["file"].filename

        file = request.args["file"][0]

        #code to upload file to back-end server goes here...

When I try this, I get an error: I/O operation on closed file.

Answer 1

You need to actually copy the file into a buffer in memory or into a tempfile on disk before you finish the request object (which is what happens when you redirect).

So you are starting your thread and handing it the request object, it's maybe opening a connection to your backend server and beginning to copy when you redirect which finishes the request and closes any associated tempfiles and you're in trouble.

Instead of passing the whole request to your thread a quick test would be trying to just pass the content of the request to your thread:

thread.start_new_thread(self.upload, (request.content.read(),))