Python, Twisted, Django, reactor.run() causing problem

Question

I have a Django web application. I also have a spell server written using twisted running on the same machine having django (running on localhost:8090). The idea being when user does some action, request comes to Django which in turn connects to this twisted server & server sends data back to Django. Finally Django puts this data in some html template & serves it back to the user.

Here's where I am having a problem. In my Django app, when the request comes in I create a simple twisted client to connect to the locally run twisted server.

...
        factory = Spell_Factory(query) 
        reactor.connectTCP(AS_SERVER_HOST, AS_SERVER_PORT, factory)
        reactor.run(installSignalHandlers=0)
        print factory.results
...

The reactor.run() is causing a problem. Since it's an event loop. The next time this same code is executed by Django, I am unable to connect to the server. How does one handle this?

Answer 1

The above two answers are correct. However, considering that you've already implemented a spelling server then run it as one. You can start by running it on the same machine as a separate process - at localhost:PORT. Right now it seems you have a very simple binary protocol interface already - you can implement an equally simple Python client using the standard lib's socket interface in blocking mode.

However, I suggest playing around with twisted.web and expose a simple web interface. You can use JSON to serialize and deserialize data - which is well supported by Django. Here's a very quick example:

import json
from twisted.web import server, resource
from twisted.python import log

class Root(resource.Resource):
    def getChild(self, path, request):
        # represents / on your web interface
        return self

class WebInterface(resource.Resource):
    isLeaf = True
    def render_GET(self, request):
        log.msg('GOT a GET request.')
        # read request.args if you need to process query args
        # ... call some internal service and get output ...
        return json.dumps(output)

class SpellingSite(server.Site):
    def __init__(self, *args, **kwargs):
        self.root = Root()
        server.Site.__init__(self, self.root, **kwargs)
        self.root.putChild('spell', WebInterface())

And to run it you can use the following skeleton .tac file:

from twisted.application import service, internet

site = SpellingSite()
application = service.Application('WebSpell')
# attach the service to its parent application
service_collection = service.IServiceCollection(application)
internet.TCPServer(PORT, site).setServiceParent(service_collection)

Running your service as another first class service allows you to run it on another machine one day if you find the need - exposing a web interface makes it easy to horizontally scale it behind a reverse proxying load balancer too.

Answer 2

reactor.run() should be called only once in your whole program. Don't think of it as "start this one request I have", think of it as "start all of Twisted".

Running the reactor in a background thread is one way to get around this; then your django application can use blockingCallFromThread in your Django application and use a Twisted API as you would any blocking API. You will need a little bit of cooperation from your WSGI container, though, because you will need to make sure that this background Twisted thread is started and stopped at appropriate times (when your interpreter is initialized and torn down, respectively).

You could also use Twisted as your WSGI container, and then you don't need to start or stop anything special; blockingCallFromThread will just work immediately. See the command-line help for twistd web --wsgi.

Answer 3

You should stop reactor after you got results from Twisted server or some error/timeout happening. So on each Django request that requires query your Twisted server you should run reactor and then stop it. But, it's not supported by Twisted library — reactor is not restartable. Possible solutions:

• Use separate thread for Twisted reactor, but you will need to deploy your django app with server, which has support for long running threads (I don't now any of these, but you can write your own easily :-)).

• Don't use Twisted for implementing client protocol, just use plain stdlib's socket module.