Putting many entities in the datastore at once

Question

I have a loop where I put a lot of entities into the datastore, something like this:

for line in open(file):
    if count >= limit:
        break
    elif count >= offset:
        prop1 = XXX
        prop2 = YYY
        MyEntity(prop1=XXX, prop2=YYY).put()
    count += 1

I know about using the bulk uploader, but I need to do this from within the code.. What I currently have works as long as the limit is not that big (otherwise I get a deadline exceeded error, but that's not the scope of this question), I am only asking if there's a better or more efficient approach for doing this, as this seems like an ugly hack, for example an approach like putting all the entities in one shot instead of looping or so.

Answer 1

You're doing a datastore round trip for each entity you store. Instead, accumulate them and do a single batch put:

to_write = []
for line in open(file):
  #...
  to_write.append(MyEntity(prop1=XXX, prop2=YYY)
db.put(to_write)

If you still need to break your operation up into multiple parts, use Task Queues to enqueue each chunk of work as a separate task.