How avoid pooling data in memory. When iterate cursor object in pymongo?
Pergunta
How avoid pooling data in memory. When iterate cursor object in pymongo?
Example:
def iter():
c=pymongo.Connection()
cursor=c.db.media.find().skip(0).limit(50000)
for item in cursor:
yield item
Before it goes in cycle for
there is pause about 2 minus. It loads all data in memory before start iterate for some reason. Can i somehow avoid it?
If I do it in mongodb shell everything is ok.
Solução
Do you know if this is possible? If c.db.media.find() returns everything instead of an iterator, I'm not sure there's much you can do.
Outras dicas
Look at cursor's block_size method. With it you should be able to set how much you read in advance. I say should, because I'm facing some problems with it now (Getting StopIteration exception on next(cursor) when modifying batch_size in pymongo), but I'm probably making some mistake. block_size should solve your problem.