Status of Python Celery tasks

Question

I'm wondering what kind of options there are for monitoring celery tasks from a browser, after they have been deployed to a worker?

My current application stack is a flask app running inside twisted, using celery to run dozens to thousands of small background tasks (updating metadata in a repository, creating image derivatives, etc.) I'm envisioning using ajax long-polling to monitor the status of the celery tasks initiated by the user. I'm using redis for the backend broker and results.

I see celery has some command line ways to monitor tasks, or flower for a web dashboard. But if I wanted to see more detailed status from a particular task sent to celery, would it make more sense for that task to print / write to a log file, then long-poll that file for changes from the flask front-end?

At this point a user can say, "update these 10,000 items", the tasks are sent to celery, and the front-end very quickly says, "job sent!". And the tasks do complete. But I'd like to have the user navigate to "/status" and see the status of those 10,000 small jobs - even a scrolling log file would probably work.

Any suggestions would be greatly appreciated. Took a lot of head scratching to make it this far sketching things out, but I'm spinning my wheels figuring out exactly WHAT to long-poll from the user front-end.

Answer 1

Jobtastic was a great idea, but not quite what worked for us. In the end, decided to create an incrementing job number (stored in Redis alongside results and broker), push all celery task id's associated with that job number into a python object, then pickle and store that in redis. We can then use that later to see if the entire "job" is complete, or the status thereof. For our purposes, works just lovely.

Answer 2

Try Jobstatic, which is extending Celery.

From project description:

Jobtastic gives you goodies like:

• Easy progress estimation/reporting
• Job status feedback
• Helper methods for gracefully handling a dead task broker (delay_or_eager and delay_or_fail)
• Super-easy result caching
• Thundering herd avoidance
• Integration with a celery jQuery plugin for easy client-side progress display
• Memory leak detection in a task run