What's the execution model of uwsgi spoolers?

Question

I need a task queue so that some of the heavy operations can be moved out of the uwsgi context without affecting the users. Since we're already using uwsgi app server, it'd be desirable if uwsgi spooler could be used as a task queue. I wanted to know how exactly it works. Are the spooled tasks still executed in some other uwsgi worker? If yes, then the server would still be overloaded since the other workers would be busy serving the spooled task. Are there better alternatives? I'm using python.

Answer 1

Reference: http://uwsgi-docs.readthedocs.org/en/latest/Spooler.html

each spooler is a different process aimed at running tasks enqueued in the form of files in a directory (the spool directory). Multiple spooler processes (per uWSGI-instance) can sit on the same spool dir to parallelize task-groups and multiple spooldirs can be configured (to have different task groups)

The spooler approach is very low-level, but requires zero-maintainance (and eventually removing tasks is a matter of rm'ing a file) and it is really solid.

The only alternative (and very probably the most used one) in the python world i am aware of is celery

http://www.celeryproject.org/

otherwise you can rely on the venerable redis + daemon thread approach, where a python thread consumes tasks enqueued in redis. Eventually you can use a uWSGI mule (it is like a worker but without external access) instead of a thread to consume tasks.

Answer 2

The uWSGI Spooler is a queue manager built into uWSGI that works like a printing/mail system. This approach is very low-level, but it is really solid.

The most popular queue for Python is Celery, which offers a wide range of task-management options and abilities. Unfortunately, Celery-based systems aren’t the easiest to keep up and running, and when something doesn’t go quite right, the issue usually isn’t that easy to find. You can ask any DevOps engineer about their experience with Celery, but be prepared for some harsh words.

Luckily, there’s an alternative: uWSGI Spooler, and that’s what we’ll be talking about in today’s article.

The main difference between uWSGI Spooler and Celery is that the former doesn’t require any additional components (Celery may need a storage system like Redis, for example), which cuts the number of points of failure in half. Tasks can be stored in a directory, external directory, or network pool.

So, there is a "must read" Reference list:

• https://uwsgi-docs.readthedocs.io/en/latest/Spooler.html
• https://blog.selectel.com/uwsgi-spooler/
• https://smirnov-am.github.io/background-jobs-with-flask/