How can I make my App Engine Datastore user search faster?

Question

I need to be able to search my users faster. A search without memcache takes 8 seconds. Now my code utilizes memcache with the help of Not Dot Net which reduces the search time to 4 sec. My question now is, how can I make it faster?

qUserSearch = Utilities.deserialize_entities(memcache.get("qUserSearch"))
if not qUserSearch:
    qUserSearch = db.GqlQuery("SELECT * FROM User ORDER BY created DESC").fetch(100000)
    memcache.add("qUserSearch", Utilities.serialize_entities(qUserSearch))

searchLength = len(searchData) 
hits = []
gotHits = False

for u in qUserSearch:
    searchEmail = u.email[0:searchLength]
    if searchEmail == searchData:
        hits.append( u.key() )
    else:
        # Since I only search for the first x chars
        # will there never be hits after it's had hits
        if gotHits:
            break 
return hits

My first ideas:

• convert to ndb and SELECT only, needed, data FROM User
• SQL way, create a table with less data. Just usernames and key to metadata
• The new full text search
• The old searchable models by 3rd party devs
• Memcache only needed properties and traverese that instead of everything

Or do you have other ideas? Which once do you think I will save the most time on?

Answer 1

You can also simulate the prefix search you are trying to accomplish with two inequality filters

db.GqlQuery("SELECT * FROM User WHERE email >= :1 AND email <= :2", searchData, unicode(searchData) + u"\ufffd")

Note: This answer was taken from the second answer provided for the question Google App Engine: Is it possible to do a Gql LIKE query?.