How can I store files on Google Storage, but skip the GAE part for fetching them?

Question

I like Google App Engine for the Datastore (and the fact that I can develop in Python) but they have 1MB limits on fetching and uploading files. On the other hand it wasn't made for that I guess.

I managed to create a service that is getting a URL (or a file from user), fetching it on GAE and then putting it on Google Storage. Works perfectly fine, but for files that are less than 1MB. There are workarounds of course, but I think it should be feasible to do it faster by skipping the GAE for data retrieval.

So my question is: Is it possible to keep my datastore on GAE and by knowing only the URL (or a special form to upload files) create the headers and then letting Google Storage to fetch the file and store it without the 1MB limit?

Answer 1

No. Google Storage doesn't support fetching the file for you.

Answer 2

It doesn't matter where the file is stored. Not only is there a 1MB datastore entity size limit, but there is a 1MB in-memory datastructure limit. So even if you stored the file somewhere else, you would have to split it into pieces to be able to operate on it on the app engine.

For Java on the GAE, I suggest looking at gaevfs (http://code.google.com/p/gaevfs/). It turns the app engine datastore into a filesystem that can store files of arbitrary size and operate on them just as normal files.