Use GAE's GQL IN filter with more than 30 items?

Question

The IN filter is limited to 30 items. Is it possible to override that limitation? Or is there a viable/recommended workaround?

Answer 1

It is not possible to override that limitation. Here's why.

When you do an IN query for where x in [n1, n2, n3], what actually happens is that 3 separate queries are submitted to the datastore:

where x = n1
where x = n2
where x = n3

The resulting lists of keys are then merged together, removing duplicates, to give you the final result.

Because each item in the in list creates a separate actual Datastore query, and there is a limit of 30 actual datastore queries per GQL query, you are limited to 30 items in the list.

Answer 2

As stated above, you can only get 30 items in a query because the query is interpreted in 30 parallel distinct queries. A way to get around is to make serial requests of 30 items like in the example below

offset = 0
item_count = len(keys)
result = []
while offset < item_count:
   data = Entity.all().filter('key IN', keys[offset:30])
   result.extend(data)
   offset += 30 
return result