Best practice for performing long computation in the background on Google App Engine?

Question

I am struggling with how best to perform a long computation for my web app. Basically I need to download and parse a very large remote dataset (~500mb, ~150mb zipped) every day.

I originally thought to just schedule it as a cron job and load a frontend instance... the problem I was thinking is that this instance is so much more computationally intensive than anything else my app does, so it seems wasteful to increase the frontend memory limit above 128mb just for this one step.

I was looking into running it as a backend, but it doesn't seem like there is a straightforward way to load a backend on a schedule like a cron job (I guess cron.yaml only loads frontend instances?)

I guess I can use task queues, but can I schedule it as a task queue and start it with cron.yaml?

I hope my questions make sense -- I appreciate any help!

Answer 1

I guess cron.yaml only loads frontend instances?

The 'target' part of the cron config can set to the name of a backend to have the code run there.

but can I schedule it as a task queue and start it with cron.yaml?

Yes. You can have a cron task (running on a frontend) that schedules a task that will run on a backend. The benefit of this over targetting the backend via cron directly, is that you can configure the retry behaviour of tasks, but can't do so for crons.