A Read-Only Relational Database on Google App Engine?

Question

I have a medium size (~100mb) read-only database that I want to put on google app engine. I could put it into the datastore, but the datastore is kind of slow, has no relational features, and has many other frustrating limitations (not going into them here). Another option is loading all the data into memory, but I quickly hit the quota imposed by google. A final option is to use django-nonrel + djangoappengine, but I'm afraid that package is still in its infancy.

Ideally, I'd like to create a read-only sqlite database that uses a blobstore as its data source. Is this possible?

Answer 1

I don't think you're likely to find anything like that...surely not over blobstore. Because if all your data is stored in a single blob, you'd have to read the entire database into memory for any operation, and you said you can't do that.

Using the datastore as your backend is more plausible, but not much. The big issue with providing a SQLite driver there would be implementing transaction semantics, and since that's the key thing GAE takes away from you for the sake of high availability, it's hard to imagine somebody going to much trouble to write such a thing.

Answer 2

django-nonrel does not magically provide an SQL database - so it's not really a solution to your problem.

Accessing a blobstore blob like a file is possible, but the SQLite module requires a native C extension, which is not enabled on App Engine.

Answer 3

While it is possible to access Blobstore objects via the BlobReader class as file-like objects, it'd probably perform even worse the the datastore to try to do relational database operations on such a file without loading the entirety of the file into memory first.