Magento 2.3 - what is the recommended way of looping through a large product collection

Question

When working with large product collections (50,000 products +) in Magento 1 I ended up using something similar to the following.

public function getProductCollection()
{
    $productsCollection = Mage::getModel('catalog/product')
        ->getCollection()
        ->addAttributeToSelect(array('name', 'price', 'visibility'));

    $productsCollection->setPageSize(50);

    $pages = $productsCollection->getLastPageNumber();
    $currentPage = 1;

    do {
        $productsCollection->setCurPage($currentPage);
        $productsCollection->load();

        foreach ($productsCollection as $_product) {

            echo $_product->getId() . "\n";
            echo $_product->getSku() . "\n";
            echo $_product->getName() . "\n";

            /* do stuff */


        }

        $currentPage++;
        //clear collection and free memory
        $productsCollection->clear();
    } while ($currentPage <= $pages);
}

Does anyone have experience on working with very large product collections in Magento2? What is the recommended approach?

Answer 1

Someone on the magento forum suggested something like this:

public function __construct( 

   \Magento\Catalog\Model\ResourceModel\Product\CollectionFactory $collectionFactory
) {
    $this->collectionFactory = $collectionFactory;
}

public function yourMethod()
{
    $collection = $this->collectionFactory->create();
    $collection->addAttributeToSelect(['name','price','visibility']);
    $collection->setPageSize(50);
    $pages = $collection->getLastPageNumber();
    for ($pageNum = 1; $pageNum<=$pages; $pageNum++) {
        $collection->setCurPage($pageNum);
        foreach ($collection as $item) {
            //Do something.
        }
        $collection->clear();
    }
}

Answer 2

Also I've come across this:

https://magento.stackexchange.com/a/213059/70343

In theory sound although I've not tested.

Answer 3

The recent comment reminded me I later came back to this and wrote two versions of an export script borrowing some logic from core for iterator

Uses iterator

https://github.com/DominicWatts/ProductCsvExport/blob/1.0.2/Console/Command/Product.php#L297

Uses collection

https://github.com/DominicWatts/ProductCsvExport/blob/1.0.1/Console/Command/Product.php#L238

Iterator handles memory usage but is much slower approach. If you've got a collections with thousands of products it might be the only suitable approach

Anyway hope helps someone