App Engine: Datastore quota reached, thou data ≈ 200 MB --?

Question

I've written an app that scans the internet and saves some data it retrieves from there. After a while the percentage of Datastore quota (Total stored data) jumped from 7% to 99%. I stopped my robot, but the figure raised to 100% after some time. The Datastore stats, though, says that the total volume of data stored in the datastore is about 200MB, the total number of entities is 501,000...

Does anyone know why can that be?

Thank you in advance.

Tim.

Answer 1

It could be indexes. If you have many indexed properties, especially list properties, the data storage number can easily be several times higher than the stats stored data number.

There is a good article explaining how space is used. http://code.google.com/appengine/articles/storage_breakdown.html

You can also star issue 2740 to request that statistics are provided for indexes too. http://code.google.com/p/googleappengine/issues/detail?id=2740