Google App Engine - Too many Datastore Read Operations

Question

I have implemented an online leaderboard via Google App Engine for my Android app. But after 2 hours I reached 100% of my quotas in "Datastore Read Operations". Can anybody help me to modify my code to reduce the read operations?
Here is my code:

public class The_Big_Bang_Theory_Quiz_HighscoreserverServlet extends HttpServlet {
public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
    String game = req.getParameter("game");
    String name = req.getParameter("name");
    String pointsStr = req.getParameter("points");
    String behaviorStr = req.getParameter("behavior");
    int behavior = 0; // 0 = upload, 1 = download
    if (behaviorStr != null) {
        try {
            behavior = Integer.parseInt(behaviorStr);
        } catch (NumberFormatException e) {
            behavior = 0;
        }
    }
    if (behavior == 0) {
        int points = 0;
        if (pointsStr != null) {
            try {
                points = Integer.parseInt(pointsStr);
            } catch (NumberFormatException e) {
                points = 0;
            }
        }
        if (points > 0 && name != null) {
            addHighscore(game, name, points);
        }
    } else {
        String maxStr = req.getParameter("max");
        int max = 1000;
        if (maxStr != null) {
            try {
                max = Integer.parseInt(maxStr);
            } catch (NumberFormatException e) {
                max = 1000;
            }
        }
        returnHighscores(resp, game, max);
    }
}

private void returnHighscores(HttpServletResponse resp, String game, int max) {
    DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
    Key gameKey = KeyFactory.createKey("game", game);
    Query query = new Query("highscore", gameKey);
    query.addSort("points", Query.SortDirection.DESCENDING);
    List<Entity> highscores = datastore.prepare(query).asList(FetchOptions.Builder.withLimit(max));
    for(Entity e : highscores) {
        try {
            resp.getWriter().println(e.getProperty("name") + ";" +e.getProperty("points"));
        } catch (IOException exc) {
            exc.printStackTrace();
        }
    }
}

private void addHighscore(String game, String name, int points) {
    DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
    Key gameKey = KeyFactory.createKey("game", game);
    Entity highscore = new Entity("highscore", gameKey);
    highscore.setProperty("name", name);
    highscore.setProperty("points", points);
    datastore.put(highscore);
}
}

I read something about the BlobStore. Is it a better method?

Answer 1

I had the same problem, i used the cache mechanism from GEA to solve the problem. Basicly the Cache is a Distributed HasMap

some code: create the Map:

try {
            cache = CacheManager.getInstance().getCacheFactory().createCache(
                    new ConcurrentHashMap<String, Category>());
        } catch (CacheException e) {
            Logger
                    .getLogger(TipsDAO.class.getName())
                    .severe(
                            "unable to cretate cache using an internal ConcurrentHashMap");
            cache = new ConcurrentHashMap<String, Category>();
        }

For every read pop you check the Map first, if you find i there you return, if you don't find it you read from the DB and put it in the Map before you return.

if (cache.containsKey(cat)) {
            return (Category) cache.get(cat);
        }
        try {
            Query query = entityManager
                    .createQuery("SELECT FROM Category WHERE name = ?1");
            query.setParameter(1, cat);
            Category temp = (Category) query.getSingleResult();
            cache.put(cat, temp);
            return temp;
        } catch (Exception e) {
            LOG.severe(e.getMessage());
            return null;
        }

For every write op to the DB you also write to the Map

cache.put(cat.getName(), cat);

Answer 2

Every entity returned by a read op requires a read operation. You have 50k read ops under free quota.

You seem pull up to 1000 high scores whenever you get your high scores. If you have > 1000 scores, then pulling scores 50 times will hit your limit.

Do your users really care about the top 1000 scores? That's your decision, but I highly doubt it. If you pulled the top 10 high scores, then you could have 100x more queries before you run out of quota.

The next step to save money is to use projection queries so your reads are using small ops instead of read ops.

Answer 3

FYI, if you enable the AppStats option of GAE, you can get a cool report that tells you exactly how long, and how expensive, each of your database transactions are.

This is pretty helpful when trying to debug your resource usage.