Twisted http server with async response where requests have to wait for data to become available or timeout

Question

I'm trying to write a simple http server that handles async requests that look in a data structure for a response or timeout:

1. Request arrives
2. While time < timeout check responseCollector for response (using requestId as the key)
3. If response, return it
4. If timeout, return a timeout message

I'm new to twisted and am wondering what the best way to do an async response is. I looked at some twisted Deferred docs and callLater but it wasn't clear to me what exactly I should be doing. Right now I run a blocking method using with deferToThread and wait for a timeout to elapse. I get a string not callable error for my deferred method:

Unhandled error in Deferred:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 497, in __bootstrap
    self.__bootstrap_inner()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 522, in __bootstrap_inner
    self.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 477, in run
    self.__target(*self.__args, **self.__kwargs)
--- <exception caught here> ---
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/threadpool.py", line 210, in _worker
    result = context.call(ctx, function, *args, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 59, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/python/context.py", line 37, in callWithContext
    return func(*args,**kw)
exceptions.TypeError: 'str' object is not callable

Here's my code:

from twisted.web import server, resource
from twisted.internet import reactor, threads
import json
import time

connectedClients = {}
responseCollector = {}

# add fake data to the collector
class FakeData(resource.Resource):
    isLeaf = True

    def render_GET(self, request):
        request.setHeader("content-type", "application/json")
        if 'rid' in request.args and 'data' in request.args:
            rid = request.args['rid'][0]
            data = request.args['data'][0]
            responseCollector[str(rid)] = data
            return json.dumps(responseCollector)
        return "{}"

class RequestHandler(resource.Resource):
    isLeaf = True

    def render_GET(self, request):
        #request.setHeader("content-type", "application/json")

        if not ('rid' in request.args and and 'json' in request.args):
            return '{"success":"false","response":"invalid request"}'

        rid = request.args['rid'][0]
        json = request.args['id'][0]

        # TODO: Wait for data to show up in the responseCollector with same rid
        # as our request without blocking other requests OR timeout
        d = threads.deferToThread(self.blockingMethod(rid))
        d.addCallback(self.ret)
        d.addErrback(self.err)

    def blockingMethod(self,rid):
        timeout  = 5.0
        timeElapsed = 0.0
        while timeElapsed<timeout:
            if rid in responseCollector:
                return responseCollector[rid]
            else:
                timeElapsed+=0.01
                time.sleep(0.01)
        return "timeout"

    def ret(self, hdata):
        return hdata

    def err(self, failure):
        return failure

reactor.listenTCP(8080, server.Site(RequestHandler()))
reactor.listenTCP(9080, server.Site(FakeData()))
reactor.run()

Make a request (doesn't return anything useful currently):

http://localhost:8080/?rid=1234&json={%22foo%22:%22bar%22}

Add some fake data to use with a request:

http://localhost:9080/?rid=1234&data=foo

Updated with working version

from twisted.web import server, resource
from twisted.internet import reactor, threads
import json
import time

connectedClients = {}
responseCollector = {}

# add fake data to the collector
class FakeData(resource.Resource):
    isLeaf = True

    def render_GET(self, request):
        request.setHeader("content-type", "application/json")
        if 'rid' in request.args and 'data' in request.args:
            rid = request.args['rid'][0]
            data = request.args['data'][0]
            responseCollector[str(rid)] = data
            return json.dumps(responseCollector)
        return "{}"

class RequestHandler(resource.Resource):
    isLeaf = True

    def render_GET(self, request):

        if not ('rid' in request.args and 'data' in request.args):
            return '{"success":"false","response":"invalid request"}'

        rid = request.args['rid'][0]
        json = request.args['data'][0]

        # TODO: Wait for data to show up in the responseCollector with same rid
        # as our request without blocking other requests OR timeout
        d = threads.deferToThread(self.blockingMethod,rid)
        d.addCallback(self.ret, request)
        d.addErrback(self.err)

        return server.NOT_DONE_YET

    def blockingMethod(self,rid):
        timeout  = 5.0
        timeElapsed = 0.0
        while timeElapsed<timeout:
            if rid in responseCollector:
                return responseCollector[rid]
            else:
                timeElapsed+=0.01
                time.sleep(0.01)
        return "timeout"

    def ret(self, result, request):
        request.write(result)
        request.finish()

    def err(self, failure):
        return failure

reactor.listenTCP(8080, server.Site(RequestHandler()))
reactor.listenTCP(9080, server.Site(FakeData()))
reactor.run()

Answer 1

In render_GET() you should return twisted.web.server.NOT_DONE_YET. You should pass the request object to ret method: d.addCallback(self.ret, request)

Then in ret(request), you should write the async data using request.write(hdata) and close the connection with request.finish().

def ret(self, result, request):
    request.write(result)
    request.finish()

Some info from: http://twistedmatrix.com/documents/current/web/howto/using-twistedweb.html

Resource rendering occurs when Twisted Web locates a leaf Resource object to handle a web request. A Resource's render method may do various things to produce output which will be sent back to the browser:

• Return a string
• Request a Deferred, return server.NOT_DONE_YET, and call request.write("stuff") and request.finish() later, in a callback on the Deferred.

Answer 2

Think about the difference between these two versions of a line of code from your example:

d = threads.deferToThread(self.blockingMethod(rid))

vs

d = threads.deferToThread(self.blockingMethod, rid)

Read the API documentation for deferToThread, and perhaps read some Python documentation about function objects (the python.org documentation doesn't cover this very well, but lots of third-party documentation does).