Reading and writing large files to and from Google Drive or Cloud Storage using Google App Engine

Question

I was wondering if Google App Engine supports reading and writing large files ( example, text files of size more than 2GB ) into and from Google Drive or Cloud Storage ?

What problems can I expect?

I'm using the Python 2,7 Google App Engine SDK by the way.

---

Updates.

I intend to read up to a million rows of data from Google's Data Store ( or maybe the new NDB ), and save the data into a text file for further processing either on Google Cloud Compute or some third party services like PiCloud.

The data is basically a network relationship and it goes like this:

A -> B
B -> C
A -> D

The above means that A is linked to B, B is linked to C and A is linked to D and so on...

As i have over a million edges, i think i might have to use the task queue or cron job to do this?

So after i processed the relationships, i'll have another text file which contains some scores between each pair of nodes, which i will than write back into the database.

Best Regards.

Answer 1

Yes it does - the question is how do you intend to write the files?

You can either upload direct to Cloud Storage using either gsutil or create_upload_url, or you can write from your app using this files API.

If you're using the file API then you need to read or write in chunks no larger than 32MB.

Answer 2

an API is around for app engine or I suppose you can use the REST cloud api. I just started with this but here is the api page https://developers.google.com/appengine/docs/python/googlestorage/