Why does subprocess.Popen block the response of SimpleHTTPServer.SimpleHTTPRequestHandler

Question

I have the strange problem that the return of a request in SimpleHTTPRequestHandler is blocked if in the request handler a new process is spawned by subprocess.Popen. But shouldn't Popen be asynchronous?

The behavior is reproducible by the following files and using Python 2.6.7 on my OS X machine as well as ActivePython 2.6 on a SLES machine: webserver.py:

#!/usr/bin/env python
import SimpleHTTPServer
import SocketServer
import subprocess
import uuid

class MyRequestHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
    def do_POST(self):
        print 'in do_POST'
        uuid_gen = str(uuid.uuid1())
        subprocess.Popen(['python', 'sleep.py'])
        print 'subprocess spawned'
        return self.wfile.write(uuid_gen)

Handler = MyRequestHandler

server = SocketServer.TCPServer(('127.0.0.1', 9019), Handler)

server.serve_forever()

sleep.py:

import time
time.sleep(10)

When I now start the webserver and then do a curl POST request to localhost:9019 the webserver prints instantly:

$python2.6 webserver2.py
in do_POST
subprocess spawned

but on the console where I do the curl request it shows the following behavior:

$curl -X POST http://127.0.0.1:9019/
<wait ~10 seconds>
cd8ee24a-0ad7-11e3-a361-34159e02ccec

When I run the same setup with Python 2.7 the answer arrives on the curl site instantly. How can this happen, since Popen doesn't seem to block the print, but just the return? The problem is I'm bound to python 2.6 for legacy reasons, so what would be the best workaround for that behavior to let the request return instantly?

Answer 1

Ok, so after Nikhil referred me to this question https://stackoverflow.com/questions/3973789/... I figured out that I need self.send_response(200), self.send_header("Content-Length", str(len(uuid_gen))) and self.end_headers().

This means the working code looks like this:

#!/usr/bin/env python
import SimpleHTTPServer
import SocketServer
import subprocess
import uuid

class MyRequestHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
    def do_POST(self):
        print 'in do_POST'
        uuid_gen = str(uuid.uuid1())
        subprocess.Popen(['python', 'sleep.py'])
        print 'subprocess spawned'
        self.send_response(200)
        self.send_header("Content-Length", str(len(uuid_gen)))
        self.end_headers()
        self.wfile.write(uuid_gen)

Handler = MyRequestHandler

server = SocketServer.TCPServer(('127.0.0.1', 9019), Handler)

server.serve_forever()

The problem seemed to be that the HTTP connection is left open for Pipelining, even though I'm still not quite sure why it the returns exactly when the Popen process is finished.