GAE: how to properly handle google shutting down a backend instance?

Question

Is there a good tutorial for how to properly write a backend/module for GAE to handle shutdowns?

This is the error I am getting:

2014-04-09 12:15:44.726 Process terminated because the backend took too long to shutdown.

I have a process that will take a few hours, and I know that I'll have to basically save the state into the memcache, and then restart it.

Are there tutorials for: 1) how to handle a shutdown request 2) how to save to memcache 3) how to restart a module

1) and 2) seem straightforward... restarting a module is something I'm unsure about. My module starts as a cron job, but is there a way to use a shutdown request to trigger another instance of my module to start?

Answer 1

If you have big jobs operating on a large amount of data, then you might look into map reduce.

Anyhow, you should break your large job down to smaller idempotent tasks. Idempotent (without side effects) basically means you can rerun a task getting the same results.

Once you have smaller tasks you can choose to schedule them via Task Queue or use a map reduce framework.

Things to note:

1. You are not guaranteed to get a shutdown callback. Our backend instances are restarted every day and our shutdown hooks do not get called.
2. Memcache is not reliable. Do not treat it as permanent storage.