Serving large generated files using Google App Engine?

Question

Presently I have a GAE app that does some offline processing (backs up a user's data), and generates a file that's somewhere in the neighbourhood of 10 - 100 MB. I'm not sure of the best way to serve this file to the user. The two options I'm considering are:

1. Adding some code to the offline processing code that 'spoofs' it as a form upload to the blob store, and going thru the normal blobstore process to serve the file.

2. Having the offline processing code store the file somewhere off of GAE, and serving it from there.

Is there a much better approach I'm overlooking? I'm guessing this is functionality that isn't well suited to GAE. I had thought of storing in the datastore as db.Text or Dd.Blob but there I encounter the 1 MB limit.

Any input would be appreciated,

Answer 1

I think storing it in the blobstore via a form post is your best currently-available option. We have plans to implement programmatic blobstore writing, but it's not ready quite yet.

Answer 2

We need to mention that since some time ago you can use the experimental feture of the blobstore to write files.

Then you can serve the file as a download using a nice BlobstoreDownloadHandler

Answer 3

I would stick with the first option. Preparing the blob will require some additional coding, but blobstore API allows for serving byte ranges of the file:

http://code.google.com/appengine/docs/python/blobstore/overview.html#Serving_a_Blob

You will not need to implement serving file chunks yourself.

Answer 4

There is some approach you are overlooking, although I'm not sure whether it is that much better:

Split the data into many 1MB chunks, and have individual requests to transfer the chunks.

This would require cooperation from the outside applications to actually retrieve the data in chunks; you might want to use the HTTP Range header to maintain the illusion of a single file. Then have another object that keeps the IDs of all the individual chunks.