Managing task queues in a multi-tenant application - GAE

Question

I have to implement recurring tasks in a multi-tenant environment on AppEngine. The real use case is 'implement recurring invoices'.

I am doubting 3 approaches, from centrally managed to decentralized per namespace to decentralized per scheduled task. Seeking feedback.

• Centrally managed : there is a 'main task queue' or cron job that periodically checks for which namespace jobs must be executed and then kicks them off. Per namespace look for tasks to be done.

• Decentralized per namespace : there is a task or cron job per namespace that periodically checks for the namespace it belongs to what needs to be done.

• Decentralized per task : when a new job is created in a namespace, immediately schedule a task queue for that task at the desired point in the future. No 'managing' task queue.

I am posing myself questions over each approach :

• Centrally managed : CONTRA -> does not scale, it requires iterating over all 'customer' namespaces. PRO -> easier to manage/control what is going on.

• Decentralized per namespace : best of both worlds ?

• Decentralized per task : PRO -> scales well, no overhead in management; CONTRA -> what when the version upgrades, what if there's a bug and scheduling did not work properly and you have to correct... sounds like a nightmare. What if scheduling fails ? Can I have a 100% garantuee that a task scheduled a year from now will get exectured ?

Is there somebody out there with good advice/experiences ?

Answer 1

You may not have that customer a year from now, or the terms might change, etc., not to mention all the technical challenges. Schedule in the future approach would not work.

I don't see any difference between the other two approaches, unless you have millions of name spaces.

I don't know how you store your data. Did you consider creating a separate table/entity for invoicing, independent of namespaces? You insert a record there, similar to your "future" approach, and create a cron job to check this table once a day/week/month. If you delete a customer or update the terms, you update this table at that time too.