Manage different stages of development with AppEngine

Question

I'm wondering : I have to manage an application on GAE, and have to manage this application at different stages of its life cycle. The app uses datastore, Google Cloud Storage, and a bit of Google Maps API. I would like to have at least two environment, development and production, if not more.

Is it possible to do so easily ? How should I do that ? For instance, do I need to have two separate apps in my appengine console ?

Also, there is a big datastore with many users (about 9000 and counting), which I need to manage, but i would like to be able to copy this datastore form an environment to another (to avoid polluting the production datastore with dev values). Is it possible? and if yes, how slow would it be to do an automatic backup of the production datastore in the dev datastore, knowing that there are about 1 000 000 entities in it?

Are there some tools that can provide help in my case (besides Git and Maven ;) ?

Answer 1

Depends whether you are testing UI (front-end) or data (back-end). Appengine supports multiple versions which access the same dDatastore, essentially different front-ends talking to the same back-end. If you are doing development which will not contaminate your data, and can be tested without contaminating your data, you can just do it on a development version. It's easy to later traffic split or set this to the default version.

If you are changing data - say, refactoring the way you store your users and entities - then you need to have a separate appengine app to develop and test that. You can backup and download the datastore from the admin console, as well as copy the data to another app, from Data -> Datastore Admin on the left. It is documented here. I don't think there is an API to do this, although it would not be much code to write. I have a script to backup / restore Text Search data (which the admin console does not support), and it's only a few hundred lines.

Answer 2

We have 2 different apps on GAE. We have different data sets in each, you can backup and restore data between apps (https://developers.google.com/appengine/docs/adminconsole/datastoreadmin). I would not recommend this as a periodic thing as it can cost you a lot of money. It will count towards your quota.

If you're looking to have the same data for 2 different env. you can use versions like user2615861 suggested above - but I would recommend against it. If you deploy a version with an entity which is not compatible with another version you can break things in your app.

I would highly recommend having 2 apps.