How avoid pooling data in memory. When iterate cursor object in pymongo?

Question

How avoid pooling data in memory. When iterate cursor object in pymongo?

Example:

def iter():
    c=pymongo.Connection()
    cursor=c.db.media.find().skip(0).limit(50000)
    for item in cursor:
        yield item

Before it goes in cycle for there is pause about 2 minus. It loads all data in memory before start iterate for some reason. Can i somehow avoid it?

If I do it in mongodb shell everything is ok.

Answer 1

Do you know if this is possible? If c.db.media.find() returns everything instead of an iterator, I'm not sure there's much you can do.

Answer 2

Look at cursor's block_size method. With it you should be able to set how much you read in advance. I say should, because I'm facing some problems with it now (Getting StopIteration exception on next(cursor) when modifying batch_size in pymongo), but I'm probably making some mistake. block_size should solve your problem.