Create, manage and kill background tasks in flask app

Question

I'm building flask web app where user can start and manage processes. These processes are doing some heavy computation (could be even days). While process is running, it saves some partial results into file to work with.

So when user starts new process, I spawn new thread and save thread handle into flask.g global variable.

def add_thread(thread_handle):
    ctx = app.app_context()
    threads = flask.g.get("threads", [])
    threads.append(thread_handle)
    g.threads = threads
    ctx.push()

Later when needed, user can terminate long running process.

def kill_thread(idx):
    ctx = app.app_context()
    threads = flask.g.get("threads", [])
    threads.pop(idx).terminate()
    g.threads = threads
    ctx.push()

I had to use ctx.push() to store list of threads, so with the next request, list is available. But this way throws exception about app context when adding new thread.

Traceback (most recent call last):
  File "/Users/mirobeka/.virtualenvs/cellular/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/mirobeka/.virtualenvs/cellular/lib/python2.7/site-packages/flask/app.py", line 1825, in wsgi_app
    ctx.auto_pop(error)
  File "/Users/mirobeka/.virtualenvs/cellular/lib/python2.7/site-packages/flask/ctx.py", line 374, in auto_pop
    self.pop(exc)
  File "/Users/mirobeka/.virtualenvs/cellular/lib/python2.7/site-packages/flask/ctx.py", line 366, in pop
    app_ctx.pop(exc)
  File "/Users/mirobeka/.virtualenvs/cellular/lib/python2.7/site-packages/flask/ctx.py", line 178, in pop
    % (rv, self)
AssertionError: Popped wrong app context.  (<flask.ctx.AppContext object at 0x10fb99c50> instead of <flask.ctx.AppContext object at 0x10fa60150>)

This starts to feel kind of wrong solution and possible dead end in later development. I was thinking about storing the handle in file/database, but lock object can't be pickled.

Is there design pattern for managing live python objects like I'm describing?

Answer 1

You better use celery (celeryproject.org) for this kind of tasks. It is heavily used in production environments, no worries of dead ends in later developments. It has all you need for managing background tasks and much much more. Here's how's to integrate it with flask.

Answer 2

Or use rq which I find 10x simpler than celery. If you run into a similar problem there - I posted my solution here.