What is the best way to implement a pool of worker processes compatible with gevent?

Question

Scenario

I have a server process implemented in Python and gevent, which is connecting to a backend service over TCP/IP and serving many Web clients (~1000 per process) with fast changing dynamic content based on the changes delivered by the backend service. The service is running on Ubuntu 10.04 and won't run on Windows, so the target platform is fixed. We're using Python 2.6.6, currently.

Problem

Serving the changes to that many clients can cause a lag in processing the changes sent by the backend, so my plan is to split the server into multiple processes. A number of worker processes would serve the Web clients, while the main process would still be connected to the backend service. I'm already using a separate pool of greenlets to serve the Web clients, but they need to be put into worker processes.

Question

Could you please point me to a working process pool implementation for gevent or figure out how can I use Python's own multiprocessing module with gevent the right way?

Restrictions

I would like to avoid introducing Python threads into our processes, since that would give room for GIL contention, which would reduce performance by introducing latencies. So it would be a clean multiprocessing + gevent solution, if possible.

Answer 1

I recommend taking a look at Celery - distributed task processing system written in Python.

The basic idea with celery is that you have a queue (either RabbitMQ or Redis), with workers written as Python processes, and exposed via the celeryd daemon. According to the celery documentation, celeryd supports running in gevent mode too (for network I/O bound processes), so presumably your worker code wouldn't need much modification to run in this environment.

Answer 2

I'd consider custom socket IPC or using ZeroMQ.