AppEngine/Python NDB key order

Question

I have an interesting problem that only showed up after upgrading my code to use NDB.

I am using gae-sessions, which encodes session expiry time into the key of each Session that is stored in the database. For example a key would have the following format: 1363968220_6ea52c936f16fc557c0c03a5f276a056 where the first 10 characters represent the timestamp of when the session expires (the value 1363968220 encodes "2013-03-22 16:03:40").

Before upgrading to NDB, the code to remove expired sessions worked correctly. After upgrading, the code erroneously eliminates sessions that have not expired. The code is the following:

time_time = time.time()
now_str = unicode(int(time_time))
now_str_datetime = datetime.datetime.fromtimestamp(time_time) 
q = SessionModel.query(namespace='')
key = ndb.Key(SessionModel, now_str + u'\ufffd', namespace='')
logging.info("Deleting sessions with key values less than %s [%s]" % (key, now_str_datetime))
q.filter(SessionModel._key < key)
results = q.fetch(500, keys_only=True)
# This is for debugging purposes
for r_key in results:
    logging.warning("Deleting key %s" % (r_key))

This results in the following output:

Deleting sessions with key values less than Key('SessionModel', '1363362698\xef\xbf\xbd') [2013-03-15 15:51:38.767947]
Deleting key Key('SessionModel', '1363924467_e0240e38b4047ef821ac62e07466351b')

If these keys are treated as strings (using string ordering/precedence rules), then the key that is deleted (starting with 1363924467) should not be considered less than the key that we are filtering against (starting with 1363362698), and so I am confused about what is going on in NDB that would cause this to not work correctly.

Answer 1

It is hard to explain, Based on the document

https://developers.google.com/appengine/docs/python/ndb/keyclass?hl=zh-tw

Keys support comparisons, for example key1 == key2 or key1 < key2. These operators compare application ID, namespace, and the full "ancestor path". They use the same ordering as the Datastore uses for queries when ordering by a key property or by key.

the cheat sheet https://docs.google.com/document/d/1AefylbadN456_Z7BZOpZEXDq8cR8LYu7QgI7bt5V0Iw/mobilebasic

and the source code. all show that the usage should be ok. The key compair the app_id, the namesapce, and the key pairs in order. Maybe you can tried to print the (key.app(), key.namespace(), key.pairs()) to see the details.

https://code.google.com/p/appengine-ndb-experiment/source/browse/ndb/key.py

def __gt__(self, other):
    """Greater than ordering."""
    if not isinstance(other, Key):
      return NotImplemented
    return self.__tuple() > other.__tuple()

def __tuple(self):
    """Helper to return an orderable tuple."""
    return (self.app(), self.namespace(), self.pairs())