What is the best way to implement a pool of worker processes compatible with gevent?
-
27-10-2019 - |
문제
Scenario
I have a server process implemented in Python and gevent, which is connecting to a backend service over TCP/IP and serving many Web clients (~1000 per process) with fast changing dynamic content based on the changes delivered by the backend service. The service is running on Ubuntu 10.04 and won't run on Windows, so the target platform is fixed. We're using Python 2.6.6, currently.
Problem
Serving the changes to that many clients can cause a lag in processing the changes sent by the backend, so my plan is to split the server into multiple processes. A number of worker processes would serve the Web clients, while the main process would still be connected to the backend service. I'm already using a separate pool of greenlets to serve the Web clients, but they need to be put into worker processes.
Question
Could you please point me to a working process pool implementation for gevent or figure out how can I use Python's own multiprocessing module with gevent the right way?
Restrictions
I would like to avoid introducing Python threads into our processes, since that would give room for GIL contention, which would reduce performance by introducing latencies. So it would be a clean multiprocessing + gevent solution, if possible.
해결책
I recommend taking a look at Celery - distributed task processing system written in Python.
The basic idea with celery
is that you have a queue (either RabbitMQ
or Redis
), with workers written as Python processes, and exposed via the celeryd
daemon. According to the celery documentation, celeryd
supports running in gevent
mode too (for network I/O bound processes), so presumably your worker code wouldn't need much modification to run in this environment.
다른 팁
I'd consider custom socket IPC or using ZeroMQ.