Consolidating Results of Tasks running on GAE (Java) Push Queue

StackOverflow https://stackoverflow.com/questions/21720439

Question

I want to put tasks onto a push queue in my GAE java app to audit the status of my users, 1 user per task. I am struggling to find a good solution for how to keep track of and consolidate the results of the audit in a safe, efficient way, across each of the tasks.

Currently what I do is use a deferred task that has one parameter in its constructor: the string representation of the key of the entity representing the user I am trying to audit.

The types of data I need to hold for each user are yes/no things like are they active, are their connections to other services revoked/not revoked, etc. When I consolidate, all I want to do is add up all the yes's and no's for each question, so e.g. I have 1000 active users, and 200 inactive users, etc.

Approaches I have considered (please excuse naivety...) are:

  • Each task puts its own entity, which I then check afterwards to consolidate the results of all the tasks
    • Pros: thread safe and safe in a distributed environment
    • Cons: relatively costly (i.e. because I have to write a lot of rows, and then later delete them), and a nuisance because I then have to post-process the entity to consolidate the data from each of the rows
  • I could grab an instance of some singleton class within each deferred task, and use that to hold and consolidate the data, and then cause it to persist its data later
    • Pros: cheap (don't have to write/delete extra data), tidy because the single object does all the pieces of work, can be made thread safe
    • Cons: on GAE I can't (easily?) rely on the singleton really being a singleton because it's distributed, so in practice I could end up with data being consolidated in two or more similar "singleton" objects
  • I suspect I could write the data into some predefined context object (the servlet context?), or memcache?
    • Pros: cheap, fast, potentially simple?
    • Cons: memcache data could get overridden - it's not safe; not sure about servlet context

Bottom line is all of these are my made up approaches, and I am sure there must be a standard approach / pattern for doing this type of thing... but I haven't been able to find it.

Any advice please...? Thanks!

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top