google app engine GQL, How to do pagination with datetime with millisecond accuracy

Question

Hi I have an issue on querying for subsecond accuracy queries from GQL.. Wondering if anyone had similar issues or workarounds.

The context to the problem is that I'm loading batches of many objects into the google data store at once. So thousands of objects could be added within a single second.

On retrieval, I run into the pagination issue when I'd like to page based on datetime of when they're added. (the last added time is ideal for paging as it allows users to get only the data that they don't have.)

Since the batches of entities are added in within a sub second quantum. The paging is problematic if I need to start paging with a batch of entities inserted in the same second.

seems app engine one box solution can't handle subsecond queries even if I try to pass in datetime objects that has fractions. i.e. I have the following query where I pass in a datetime object with fractions of second. as locally i'm running python 2.6.

test = GqlQuery("select * from table where lastupdated > :1", minimumTime)

If the minimumTime is something like 10:00:00.0500, the result set could still contain records with lastUpdated at 10:00:00.0100, 10:00:00.0200 etc..

I'm thinking maybe the workaround is to create my own key format, which needs to be autoincrementing. But just wanted to see if there was any other simpler solutions i'm missing.

Thanks, Ning

Answer 1

Don't paginate on the DateTime; instead, use cursors.