Is there a way to implement long-running transactions in App Engine?

Question

I have a function that makes many HTTP requests to collect data and stores it in the datastore. All calls must succeed or else everything must be rolled back. The length of time to execute approaches 10 minutes. The max transaction deadline on app engine appears to be the default 60 seconds.

Is there a way to increase this and thus make the function transactional?

Answer 1

It looks like it's not possible. From the docs:

Maximum time, in seconds, to wait for Datastore to return a result before aborting with an error. Accepts either an integer or a floating-point value. Cannot be set higher than the default value (60 seconds), but can be adjusted downward to ensure that a particular operation fails quickly (for instance, to return a faster response to the user, retry the operation, try a different operation, or add the operation to a task queue).

https://developers.google.com/appengine/docs/python/datastore/functions#create_transaction_options

However you can trigger a task when and only when a transaction completes successfully, if that's of any use.

https://developers.google.com/appengine/docs/python/datastore/transactions#Transactional_Task_Enqueuing

You can enqueue a task as part of a Datastore transaction, such that the task is only enqueued—and guaranteed to be enqueued—if the transaction is committed successfully. If the transaction does not get committed, the task is guaranteed not to be enqueued. If the transaction does get committed, the task is guaranteed to be enqueued.

Answer 2

In frontend instances no.

You will need to spin a backend instance for that: https://developers.google.com/appengine/docs/python/backends/overview

Answer 3

For 10 minute requests you will need to use a backend

Backends (Python)

This reference describes how to use App Engine Backends in Python. Backends are special App Engine instances that have no request deadlines, higher memory and CPU limits, and persistent state across requests. They are started automatically by App Engine and can run continously for long periods. Each backend instance has a unique URL to use for requests, and you can load-balance requests across multiple instances.

Backends can handle HTTP requests from users or other parts of your application, start and run continuously in the background, or be driven by Task Queue tasks or Cron jobs. The link above describes how to configure backends, best practices for using them, and how App Engine bills for them.