To shard or not to shard? GAE/java/jdo

Question

I'm currently porting some work from MySQL to Google App Engine/Java. I'm using JDO, as well as the lower level java API where required.

I read through the optimization guide about sharding counters: http://code.google.com/appengine/articles/sharding_counters.html

I'm still building the foundation of my app. I know that premature optimization is the root of all evil; but this is clearly documented in order to avoid contention. So I'm having trouble deciding if I should be biased one way or the other.

So should I be sharding counters (and other possibly higher frequency write operation objects) by default, or should I go forward without sharding and implement on an as needed basis?

Answer 1

The salient meaning of "premature" here is "before the proper time." Designing to avoid limits, when those limits are well understood, is not premature.

Shard your counters.

Answer 2

Even with effective sharding, maintaining aggregates can add some substantial load to your application. If you need that aggregate, and you can't afford an approximation; then using a sharded aggregate is not a premature optimization; there is no next best alternative. If you don't actually need the counter, then the time it will take to implement it could be better spent elsewhere.