Sustainable Query Per Seconds [QPS] without incurring in timeout errors

Question

Do you know which is the safest Queries Per Second rate in update on a single entity without incurring in write contention?
Reading this document about sharding it clearly says:

it is important to note that you can only expect to update any single entity or entity group about five times a second. That is an estimate and the actual update rate for an entity is dependent on several attributes of the entity, including how many properties it has, how large it is, and how many indexes need updating.

I'm having timeout problem even with one update per second and a comment in this answer has baffled me:

You can still get write contention on individual entities if you're doing more than about 1QPS of modifications to them

Am I missing something?
How could my GAE app scale if I can't even update a single entity per second without incurring in Timeout errors?

Answer 1

The reason for the variance is because this isn't a hard limit. If you do 1 QPS of updates to an entity group, you probably won't notice any significant increase in timeouts or latency at all. If you do 5 QPS, you'll probably see an increased rate of timeouts, and noticeably higher latency. Beyond that, contention issues will get worse.

Generally, I think of 1 QPS as the design goal to build around, to give a reasonable margin of error for spikes.