How to configure Task Queues for Datastore Admin to minimize frontend instance count?

Question

I am using the Datastore Admin cron job to create daily backups of my application.

cron.xml:

<cronentries>
  <cron>
    <url>/_ah/datastore_admin/backup.create?[...]queue=backup</url>
    <description>Daily Backup</description>
    <schedule>[...]</schedule>
    <timezone>America/New_York</timezone>
    <target>ah-builtin-python-bundle</target>
  </cron>
</cronentries>

I created a task queue specifically for the backups so that the performance of my default queue is not affected during the backup.

queue.xml:

<queue-entries>
  <queue>
    <name>backup</name>
    <rate>30/s</rate>
    <bucket-size>25</bucket-size>
    <max-concurrent-requests>128</max-concurrent-requests>
  </queue>
</queue-entries>

GAE launches a large number of instances during the backup process. As a result the backups finish in a few minutes. There is no need for the backup process to finish this quickly. I would like to launch fewer instances to reduce costs as long as the backups still finishes in a few hours.

How can I change the queue configuration to launch fewer instances? How can I figure out which values I should used for rate,bucket size, max-concurrent-requests so that GAE launches fewer instances but still finishes the backups in less than 24h? I reduced rate from 50 to 30 and max-concurrent-requests from 256 to 128. Both changes had no measurable impact on the instance count or the backup execution time.

Answer 1

Your queue will typically be throttled by either rate & bucket or by max-concurrent-requests.

For your backup process to finish in hours instead of minutes, you will need to reduce values drastically, for example by a factor of 15.

You haven't mentioned how long your tasks take to complete, or how many are running concurrently. Suppose some of your tasks take seconds to complete, and others take minutes. Then max-concurrent-requests will be more useful that rate & bucket in directly controlling the maximum number of instances running.