Magento 2: how to use the collection walk iterator?

Question

Back in Magento 1, it was possible to use the collection iterator to walk through the results and avoid looping through them.

It was a huge improvement in terms of performance when dealing with massive collections.

Here is some sample code of what could be done.

$customers = Mage::getModel('customer/customer')->getCollection()->addAttributeToSelect(array('firstname'), 'inner');
// call iterator walk method with collection query string and callback method as parameters
Mage::getSingleton('core/resource_iterator')->walk($customers->getSelect(), array(array($this, 'customerCallback')));

Then you could define a callback function customerCallback to process the results one by one.

Is that still possible in Magento 2? If so how can I achieve that?

Answer 1

it still seems to be there: https://github.com/magento/magento2/blob/2.0/lib/internal/Magento/Framework/Model/ResourceModel/Iterator.php

Usage should be the same as in M1

Answer 2

Yes, it's possible! It's very similar to Magento 1, by the way.

\Magento\Framework\Model\ResourceModel\Iterator has the walk function which will call your callback function for every single item, rather than the entire collection at once.

public function iterator()
{
    // you can filter by attribute
    // $this
    //    ->collection // \Magento\Catalog\Model\ResourceModel\Product\Collection
    //    ->addAttributeToFilter('name', ['like' => '%samsung%']);

    $this
        ->iterator // \Magento\Framework\Model\ResourceModel\Iterator
        ->walk(
            $this->collection->getSelect(),
            [[$this, 'callback']]
        );
}

public function callback($args)
{
    // load of the product
    $product = $this
        ->productRepository // \Magento\Catalog\Api\ProductRepositoryInterface
        ->getById($args['row']['entity_id']);

    // do whatever you want to with the $product
    print_r($product->debug());
}

I also have written a blog post about working with large collections in Magento 2. And here are an example project.

Answer 3

<?php

use Magento\Framework\Model\ResourceModel\Iterator;
use Magento\Framework\Model\ResourceModel\IteratorFactory;
use Magento\Sales\Model\ResourceModel\Order\CollectionFactoryInterface;
use Zend_Db_Select;

class RunSomeLongTask
{
    public const ORDER_STATE_VALUE = 'example';

    /** @var CollectionFactoryInterface  */
    private $orderCollectionFactory;

    /** @var IteratorFactory */
    private $iteratorFactory;

    /** @var SomeTask */
    private $someTask;

    /**
     * constructor.
     * @param CollectionFactoryInterface $orderCollectionFactory
     * @param IteratorFactory $iteratorFactory
     * @param SomeTask $someTask
     */
    public function __construct(
        CollectionFactoryInterface $orderCollectionFactory,
        IteratorFactory $iteratorFactory,
        SomeTask $someTask
    ) {
        $this->orderCollectionFactory = $orderCollectionFactory;
        $this->iteratorFactory = $iteratorFactory;
        $this->someTask = $someTask; // "important: remember create this class"
    }

    public function process(): void
    {
        $orderCollection = $this->orderCollectionFactory->create();

        $orderCollection
            ->getSelect()
            ->reset(Zend_Db_Select::COLUMNS)
            ->columns(['entity_id', 'state', 'status'])
            ->where('state=?', self::ORDER_STATE_VALUE);

        /** @var Iterator $iterator */
        $iterator = $this->iteratorFactory->create();

        $iterator->walk($orderCollection->getSelect(), [[
            $this->someTask, // instance of class SomeTask
            'run' // the function in your class "SomeTask"
        ]]);
    }
}

class SomeTask
{
    public function run(array $args): void
    {
        // you will receive the row data here
        
        $orderId = $args['row']['entity_id'];

        // do your task here on individual row
    }
}

Explained well on http://www.rosenborgsolutions.com/collection-walk-iterator.php