Is it possible to iterate over Magento collections with pagination natively?

Question

What I mean by that is - is there a way to do:

$collection = $model->getCollection();
foreach ($collection as $item) { 
    $item->doStuff();
}

In such a way that even if the collection had 100k rows, it would only load a page of rows at a time from MySQL and magically paginate them for you behind the scenes.

From looking at Varien_Data_Collection_Db::load() it doesn't seem like it's possible, but just wanted to check. This seems like something that should be a common need.

Answer 1

You should really use

Mage::getSingleton('core/resource_iterator')

for this purpose, as it exists purely for the performance reasons you mentioned.

Otherwise, you can use a slightly less elegant solution using a loop with setPageSize - there's a good example here, https://stackoverflow.com/questions/3786826/how-to-loop-a-magento-collection

Answer 2

I agree with Ben Lessani that you should use the core/iterator resource model to load big collections one row at a time if possible.

However, there are limitations. As explained in "addAttributeToSelect not working with core/resource_iterator?" it does not work well with EAV models if you need to include values from the attribute value tables.

And the linked example from StackOverflow is actually not that good because it repeats the same query with different LIMIT expressions. For complex queries this might be a performance problem, but even more important, you will get duplicates if new rows are added in between.

A better way to handle collections in chunks is to first load all the ids, then use these ids as a filter for the actual paged collection.

Simple example for products:

$ids = Mage::getModel('catalog/product')
    ->getCollection()
    ->getAllIds();

$page = 1;
do {
    $collection = Mage::getModel('catalog/product')
        ->getCollection()
        ->addIdFilter($ids)
        ->setPageSize(100)
        ->setCurPage($page);

    $results = $collection->load();

    // do stuff ......

    $page++;

} while ($results->count());