storage service with google app engine

Question

I've been assigned a project to create a file storage service on Google App Engine.But I am really doubtful whether it's possible or not given the 30 sec limit to process the response and moreover its the BigTable is just a database system not a storage server .Am I correct ?

Answer 1

You will want to look at the blobstore API

http://code.google.com/appengine/docs/java/blobstore/

It lets you upload files upto 50Mb in size and serve them directly, without going through BigTable. What I am doing is I have a script that takes in a file, creates the blobstore upload url, then post the file it received to that url and stores an entry in bigtable. The 30 second time limit only applies to processing time so the act of uploading a file does not count towards that.

Answer 2

If all you need is storage, a better choice than App Engine is the newly beta-d Google Storage:

Google Storage for Developers is a RESTful service for storing and accessing your data on Google's infrastructure. The service combines the performance and scalability of Google's cloud with advanced security and sharing capabilities. Highlights include:

Fast, scalable, highly available object store

All data replicated to multiple U.S. data centers

Read-your-writes data consistency

Objects of hundreds of gigabytes in size per request with range-get support

Domain-scoped bucket namespace

Easy, flexible authentication and sharing

Key-based authentication

Authenticated downloads from a web browser

Individual- and group-level access controls

Answer 3

Use and look if you like blobstore in action I maintain

Answer 4

This is definitely possible.

There are also some similar questions that have been answered. See Google for more implementations.

But in general, yes, the 30 second limit does complicate your task, but you can break your file up into chunks and store them each separately as text/blobs. App Engine's Remote API might do this for you automatically.

Answer 5

Have in mind that you have 10MB per request restriction.

Answer 6

Have a look at this project for some inspiration:

http://django-filetransfers.appspot.com/

the sourcecode is available if you follow a link on the page.