mapping encoded keys to shorter identifiers in appengine

Question

I want to send unique references to the client so that they client can refer back to specific objects. The encoded keys appengine provides are sometimes 50 bytes long, and I probably only need two or three bytes (I could hope to need four or five, but that won't be for a while!).

Sending the larger keys is actually prohibitively expensive, since I might be sending 400 references at a time.

So, I want to map these long keys to much shorter keys. An obvious solution is to store a mapping in the datastore, but then when I'm sending 400 objects I'm doing 400 additional queries, right? Maybe I mitigate the expense by keeping copies of the mappings in memcache as well. Is there a better way?

Can I just yank the number out of the unencoded keys that appengine creates and use that? I only need whatever id I use to be unique per entity kind, not across the whole app.

Thanks,

Riley

Answer 1

Datastore keys include extra information you don't need - like the app ID. So you definitely do not need to send the entire keys.

If these references are to a particular Kind in your datastore, then you can do even better and just send the key_name or numeric ID (whichever your keys use). If the latter is the case, then you could transmit each key with just a few bytes (you could opt for either a variable-length or fixed-length integer encoding depending on which would be more compact for your specific case [probably the former until most of the IDs you're sending get quite large]).

When you receive these partial keys back from the user, it should be easy to reconstruct the full key which you need to retrieve the entities from the datastore. If you are using the Python runtime, you could use db.Key.from_path(kind_name, numeric_id_or_key_name).

A scheme like this should be both simpler and (a lot) faster than trying to use the datastore/memcache to store a custom mapping.

Answer 2

You don't need a custom mapping mechanism. Just use entity key names to store your short identifier :

entity = MyKind(key_name=your_short_id)
entity.put()

Then you can fetch these short identitiers in one query :

keys = MyKind.all(keys_only=True).filter(...).fetch(400)
short_ids = [key.name() for key in keys]

Finally, use MyKind.get_by_key_name(short_id) in order to retrieve entities from identifiers sent back by your users.