GAE & datastore - filter and delete entities with python

Question

I'm trying to delete from a datastore table all the entities that have the type field value equals to "TEST".

Actually my table structure is something like this:

table TEST

id    |    type    |    value    
---------------------------------
1     |   TEST     |    aksjdh
2     |   FOO      |    wer
3     |   TEST     |    gg
4     |   TEST     |    werqwer
5     |   BAR      |    akvcvcxjdh

I'm able to delete ALL the entities with this piece of python code, called via an URL request:

import cgi
import datetime
import urllib
import webapp2
from google.appengine.ext import db

class Test(db.Model):
    id = db.StringProperty()
    type = db.StringProperty()
    value = db.StringProperty()

class TestHandler(webapp2.RequestHandler):
    def get(self):
        ...
        db.delete(Test.all())

app = webapp2.WSGIApplication([
    ('/deleteTest', TestHandler)
], debug=True)

What I'd like to know is: as soon as in the previous code, to delete all the Test's entities, I've used the method all() of the Test's class, do I have to use the same logic also to delete only a subset of those entities, based on their type value? Should I store the entire list in a Test object and then proceed with further filtering?

In other words, can I do

db.delete(db.GqlQuery("SELECT * FROM TEST WHERE type = :1", "TEST")) 

or is better doing something like

#I don't know if something like this exists, it's just an example
testList = Test.all()
db.delete(testList.filter("type", "TEST"))

note that in the 2nd example, I've firstly instantiated some sort of Test object, and then deleted the entities starting from that object.

I hope you get what I mean, this thing is tricky also in my mind and I'm not sure I explained it well..

Thanks in advance, best regards

Answer 1

Your GQL is OK, though you should do a keys only query as it is more efficient.

db.delete(db.GqlQuery("SELECT __key__ FROM TEST WHERE type = :1", "TEST"))

Using a filter it should be

testList = Test.all(keys_only=True)
db.delete(testList.filter("type = ", "TEST"))

You will find that for large numbers of records in production this approach won't scale. Trying to do this in a web request with many thousands of entities will more than likely end up with a DeadlineExceededError.

If you run the delete in a task you will get 10 minutes to complete your request, if the you really large numbers even this won't work.

If you are just trying to clear the dev datastore use the command line for clearing the datastore on dev server startup.

Some other things to consider.

If you are just starting out, switch to ndb. You should protect such a web method with with some sort of authentication/access control. Consider learning about REST and methodology. Normally requests that mutate shouldn't be done with a GET, but use DELETE or POST. - just some things to consider.