Python Threads (or their equivalent) on Google Application Engine Workaround?

Question

I want to make a Google App Engine app that does the following:

1. Client makes an asynchronous http request
2. Server starts processing that request
3. Client makes ajax http requests to get progress

The problem is that the server processing (step #2) may take more than 30 seconds.

I know that you can't have threads on Google Application Engine and that all tasks must complete within 30 seconds or they get shut down. Is there some way to work around this?

Also, I'm using python-django as a backend.

Answer 1

You'll want to use the Task Queue API, probably via deferred tasks. The deferred API makes working with Task Queues dramatically simpler.

Essentially, you'll want to spawn a task to start the processing. That task should catch DeadlineExceeded exceptions and reschedule itself (again via the deferred API) to continue processing. This requires that your tasks be able to keep track of their own progress. They can also update their own status in memcache, which you can use to write a view that checks a task's status. That view can then be polled via Ajax.