Django-Celery - Work in parallel on repetitive requests

Question

I am not familiar with Djano-Celery, so I would like to know if it is the right tool for what I need to do before going deeper in the doc.

My django app has a web service for tiling map images that is called like this: http://host.com/tiling/x/y/z.png

xyz are integer variables that are used in the tiling function to compute the output.

My question is: do Djano-Celery can create workers for parallel processing on this tiling function when repetitive requests are detected?

For instance, 10 or more requests could be sent by a user at a time : http://host.com/tiling/0/1/1.png, http://host.com/tiling/1/0/1.png etc...

Can Django-Celery creates workers for each in parralel instead of computing each request one by one? What are the requierements on server side? Do I need something linke NGINX or GUNICORN or WSGI or CGI? I am confuse about those things...

Answer 1

In most cases celery is used for asynchronous tasks handling. But it works also for concurrent tasks!

By default celery uses multiprocessing but you can also use Eventlet - concurrent networking library for Python.

Reference:

• http://docs.celeryproject.org/en/latest/userguide/concurrency/eventlet.html#concurrency-eventlet
• http://docs.celeryproject.org/en/latest/userguide/workers.html#concurrency