How do I run background threads after the request has been respnded with nginx/flup?

Question

My stack is Nginx + Flup + Flask.

So I have a finite thread that runs a background thread that has a finite lifetime.

t = threading.Thread(target=campaign.run, args=(campaign_obj,))
t.setDaemon(False)
t.start()

It gets executed, lasts about a minute or 2, then it ends. The thing is, for responsiveness, I have it return a value to the request first while the background process is still being run.

The problem is, somehow, somewhat, the fcgi thread keeps getting killed.

I have tried changing the connection_timeout to 1800s

    keepalive_timeout 1800s;

Restarted nginx, and now it gets cut off almost immediately after the thread is responded.

How can I either

1) Simple fix my python code / fcgi code (without refactoring into a huge new stack with redis/rabbitmq/celery as this is really a quick project)

2) Stop nginx from killing my thread, at least not so soon.

Thanks!

Answer 1

Use multiprocessing

As soon as you return a value, and complete the request, FastCGI is going to clean up after you, and that means all the threads you spawned. If you really NEED your task to go on ticking, you have to use multiprocessing so that the process you spawned can survive the end of the request.

Otherwise, Celery is really easy to set up, and solves exactly this problem your having...