How to spawn long-running-processes in Twisted

Question

Preface

I'm writing an web-server that gives to users an access to some program written on C (I'm using an Python wrapper over this C-program, it is PyCLIPS). To serve a lot of users, the web-server has to start a lot of copies of this C-program, because one copy can serve very few users in the same time, about 1-3 users. In addition, each user should work only with his own copy, therefore should be a lot of copies of the C-program.

This C-program is a CLIPS engine if it'll help to understand.

---

So, to solve this design problem, I want write a Twisted TCP server that will be like a pool of long-running-processes. Each of long-running-processes is a small Twisted TCP server that gives an access to one copy of the C-program.

In example, a user ask the pool-server to reserve a long-running-process for him, then the pool-server create and run a long-running-process that starts listening on some port, then the pool-server return the host and port of this long-running-process to user. Now, user can communicate with this long-running-process directly.

---

Questions

1. How start these long-running-process from the pool-server? The pool-server and each of long-running-processes are should be separate Twisted servers.
2. Is Twisted a good choice for these aims?
3. Maybe there are another ways how solve this design problem?

Thanks a lot.

Answer 1

Using Twisted for this sounds like it makes sense. The low-level API that supports running processes in Twisted is reactor.spawnProcess. Some examples of its usage are given in the process howto document.

This API is well suited for dealing with many long running processes, and as with all I/O APIs in Twisted, it works well when combined with another event source, such as a web server users can use to request new processes be started up.

reactor.spawnProcess is more like the standard library subprocess module than like the multiprocessing package. It gives you a way to launch a child process running a particular executable, with particular arguments, etc. It does not provide a high-level API for running a particular Python function in another process. However, it's not too hard to build such a thing (at least for a particular case). Consider this approach:

from sys import executable
from os import environ

from twisted.internet import reactor

implementation = """\
from yourapp import somefunction
somefunction()
"""

reactor.spawnProcess(executable, [executable, "-c", implementation], env=environ)
reactor.run()

This just launches a new Python interpreter (whichever one you happen to be running) and uses the -c option to specify a program on the command line for it to run.