Collection pagination skipping records

magento.stackexchange https://magento.stackexchange.com/questions/13095

Question

I'm having trouble with Magento's built-in collection paging skipping over many records. I am attempting to optimize a product feed generator that dumps a bunch of product data to a file, and have been banging my head against the wall for several days now trying to figure this one out. The code below works great if I set the page size to a value larger than the number of records returned. In other words, it works if there's only 1/1 pages to process. It does not work if it has multiple pages to process. By "does not work", I mean that many products are skipped or not returned by the Collection. Relevant code is below.

<?php
public function create()
{
    $productCollection = $this->_getProductCollection();
    $pages = $productCollection->getLastPageNumber();
    $pageNumber = 1;
    do {
        $productCollection->setCurPage($pageNumber);
        $productCollection->load();
        foreach ($productCollection as $product) {
            // Process product information and write it to file....
        }
        $pageNumber++;
        $productCollection->clear();

    } while ($pageNumber <= $pages);
}

protected function _getProductCollection()
{
    $adapter = $this->_getReadAdapter();
    $urlPathAttribute = $this->_getAttributeFromCode('url_path');
    $storeId = $adapter->quote($this->_getDefaultStoreId());

    /** @var Mage_Catalog_Model_Resource_Product_Collection $productCollection */
    $productCollection = Mage::getModel('catalog/product')->getResourceCollection();
    $productCollection->addFieldToFilter('status', Mage_Catalog_Model_Product_Status::STATUS_ENABLED)
        ->addAttributeToSort('name', Varien_Data_Collection::SORT_ORDER_ASC)
        ->addAttributeToSelect(array(
                'color', 'description', 'gender', 'height', 'image', 'manufacturer', 'manufacturerspartnumber',
                'manufacturersstylenumber', 'manufacturersupc', 'name', 'outer_material', 'oversize', 'price',
                'primary_color', 'short_description', 'size', 'special_from_date', 'special_price', 'special_to_date',
                'url_path', 'weight', 'width'
            )
        )
        ->joinField('parent_id', 'catalog/product_super_link', 'parent_id', 'product_id=entity_id', null, 'left')
        ->joinField('parent_sku', 'catalog/product', 'sku', 'entity_id=parent_id', null, 'left')
        ->setFlag('require_stock_items', 1)
        ->setPageSize($this->_page_size);

    // Add parent's url_path to child row, since simple child products are not directly visible.
    $select = $productCollection->getSelect();
    $select->joinLeft(array(
        'at_parent_url_path' => $urlPathAttribute->getBackendTable()),
        "(at_parent_id.parent_id = at_parent_url_path.entity_id) AND (at_parent_url_path.attribute_id = {$urlPathAttribute->getId()}) AND (at_parent_url_path.store_id = {$storeId})",
        array('parent_url_path' => 'at_parent_url_path.value')
    );

    return $productCollection;
}

I added a couple of debug lines to the log to illustrate what is happening:

2014-01-13T20:12:56+00:00 DEBUG (7): Started creating feed.
2014-01-13T20:12:56+00:00 DEBUG (7): START Processing product page #1/36. Current memory usage: 122.25 MB
2014-01-13T20:13:03+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:03+00:00 DEBUG (7): END Processing product page #1/36. Current memory usage: 126.75 MB
2014-01-13T20:13:03+00:00 DEBUG (7): START Processing product page #2/36. Current memory usage: 126.75 MB
2014-01-13T20:13:09+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:09+00:00 DEBUG (7): END Processing product page #2/36. Current memory usage: 126.75 MB
2014-01-13T20:13:09+00:00 DEBUG (7): START Processing product page #3/36. Current memory usage: 126.75 MB
2014-01-13T20:13:15+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:15+00:00 DEBUG (7): END Processing product page #3/36. Current memory usage: 126.75 MB
2014-01-13T20:13:15+00:00 DEBUG (7): START Processing product page #4/36. Current memory usage: 126.75 MB
2014-01-13T20:13:21+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:21+00:00 DEBUG (7): END Processing product page #4/36. Current memory usage: 126.75 MB
2014-01-13T20:13:21+00:00 DEBUG (7): START Processing product page #5/36. Current memory usage: 126.75 MB
2014-01-13T20:13:26+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:26+00:00 DEBUG (7): END Processing product page #5/36. Current memory usage: 126.75 MB
2014-01-13T20:13:26+00:00 DEBUG (7): START Processing product page #6/36. Current memory usage: 126.75 MB
2014-01-13T20:13:32+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:32+00:00 DEBUG (7): END Processing product page #6/36. Current memory usage: 126.75 MB
2014-01-13T20:13:32+00:00 DEBUG (7): START Processing product page #7/36. Current memory usage: 126.75 MB
2014-01-13T20:13:38+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:38+00:00 DEBUG (7): END Processing product page #7/36. Current memory usage: 126.75 MB
2014-01-13T20:13:38+00:00 DEBUG (7): START Processing product page #8/36. Current memory usage: 126.75 MB
2014-01-13T20:13:45+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:45+00:00 DEBUG (7): END Processing product page #8/36. Current memory usage: 126.75 MB
2014-01-13T20:13:45+00:00 DEBUG (7): START Processing product page #9/36. Current memory usage: 126.75 MB
2014-01-13T20:13:51+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:51+00:00 DEBUG (7): END Processing product page #9/36. Current memory usage: 126.75 MB
2014-01-13T20:13:51+00:00 DEBUG (7): START Processing product page #10/36. Current memory usage: 126.75 MB
2014-01-13T20:13:57+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:57+00:00 DEBUG (7): END Processing product page #10/36. Current memory usage: 126.75 MB
2014-01-13T20:13:57+00:00 DEBUG (7): START Processing product page #11/36. Current memory usage: 126.75 MB
2014-01-13T20:14:03+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:03+00:00 DEBUG (7): END Processing product page #11/36. Current memory usage: 126.75 MB
2014-01-13T20:14:03+00:00 DEBUG (7): START Processing product page #12/36. Current memory usage: 126.75 MB
2014-01-13T20:14:09+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:09+00:00 DEBUG (7): END Processing product page #12/36. Current memory usage: 126.75 MB
2014-01-13T20:14:09+00:00 DEBUG (7): START Processing product page #13/36. Current memory usage: 126.75 MB
2014-01-13T20:14:15+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:15+00:00 DEBUG (7): END Processing product page #13/36. Current memory usage: 126.75 MB
2014-01-13T20:14:15+00:00 DEBUG (7): START Processing product page #14/36. Current memory usage: 126.75 MB
2014-01-13T20:14:22+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:22+00:00 DEBUG (7): END Processing product page #14/36. Current memory usage: 126.75 MB
2014-01-13T20:14:22+00:00 DEBUG (7): START Processing product page #15/36. Current memory usage: 126.75 MB
2014-01-13T20:14:28+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:28+00:00 DEBUG (7): END Processing product page #15/36. Current memory usage: 126.75 MB
2014-01-13T20:14:28+00:00 DEBUG (7): START Processing product page #16/36. Current memory usage: 126.75 MB
2014-01-13T20:14:34+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:34+00:00 DEBUG (7): END Processing product page #16/36. Current memory usage: 126.75 MB
2014-01-13T20:14:34+00:00 DEBUG (7): START Processing product page #17/36. Current memory usage: 126.75 MB
2014-01-13T20:14:40+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:40+00:00 DEBUG (7): END Processing product page #17/36. Current memory usage: 126.75 MB
2014-01-13T20:14:40+00:00 DEBUG (7): START Processing product page #18/36. Current memory usage: 126.75 MB
2014-01-13T20:14:47+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:47+00:00 DEBUG (7): END Processing product page #18/36. Current memory usage: 126.75 MB
2014-01-13T20:14:47+00:00 DEBUG (7): START Processing product page #19/36. Current memory usage: 126.75 MB
2014-01-13T20:14:49+00:00 DEBUG (7): Processed 251 product(s).
2014-01-13T20:14:49+00:00 DEBUG (7): END Processing product page #19/36. Current memory usage: 126.75 MB
2014-01-13T20:14:49+00:00 DEBUG (7): START Processing product page #20/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:50+00:00 DEBUG (7): END Processing product page #20/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): START Processing product page #21/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:50+00:00 DEBUG (7): END Processing product page #21/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): START Processing product page #22/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:50+00:00 DEBUG (7): END Processing product page #22/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): START Processing product page #23/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:50+00:00 DEBUG (7): END Processing product page #23/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): START Processing product page #24/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:50+00:00 DEBUG (7): END Processing product page #24/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): START Processing product page #25/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:51+00:00 DEBUG (7): END Processing product page #25/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): START Processing product page #26/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:51+00:00 DEBUG (7): END Processing product page #26/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): START Processing product page #27/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:51+00:00 DEBUG (7): END Processing product page #27/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): START Processing product page #28/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:51+00:00 DEBUG (7): END Processing product page #28/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): START Processing product page #29/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:51+00:00 DEBUG (7): END Processing product page #29/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): START Processing product page #30/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:52+00:00 DEBUG (7): END Processing product page #30/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): START Processing product page #31/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:52+00:00 DEBUG (7): END Processing product page #31/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): START Processing product page #32/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:52+00:00 DEBUG (7): END Processing product page #32/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): START Processing product page #33/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:52+00:00 DEBUG (7): END Processing product page #33/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): START Processing product page #34/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:52+00:00 DEBUG (7): END Processing product page #34/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): START Processing product page #35/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:52+00:00 DEBUG (7): END Processing product page #35/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): START Processing product page #36/36. Current memory usage: 126.75 MB
2014-01-13T20:14:53+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:53+00:00 DEBUG (7): END Processing product page #36/36. Current memory usage: 126.75 MB
2014-01-13T20:14:53+00:00 DEBUG (7): Finished creating feed.

EDIT: I tried simplifying my collection with the same end result.

protected function _getProductCollection()
{
    /** @var Mage_Catalog_Model_Resource_Product_Collection $productCollection */
    $productCollection = Mage::getModel('catalog/product')->getResourceCollection();
    $productCollection->addFieldToFilter('status', Mage_Catalog_Model_Product_Status::STATUS_ENABLED)
        ->addAttributeToSort('name', Varien_Data_Collection::SORT_ORDER_ASC)
        ->addAttributeToSelect(array(
                'color', 'description', 'gender', 'height', 'image', 'manufacturer', 'manufacturerspartnumber',
                'manufacturersstylenumber', 'manufacturersupc', 'name', 'outer_material', 'oversize', 'price',
                'primary_color', 'short_description', 'size', 'special_from_date', 'special_price', 'special_to_date',
                'url_path', 'weight', 'width'
            )
        )
        ->setPageSize($this->_page_size);

    return $productCollection;
}
Was it helpful?

Solution

I was able to workaround this issue by clearing the product collection and recreating my product collection object between each page, and then setting the current page on each new collection.

<?php
public function create()
{
    $productCollection = $this->_getProductCollection();
    $pages = $productCollection->getLastPageNumber();
    $pageNumber = 1;
    do {
        $productCollection->setCurPage($pageNumber);
        $productCollection->load();
        foreach ($productCollection as $product) {
            // Process product information and write it to file....
        }
        $pageNumber++;
        $productCollection->clear();
        $productCollection = $this->_getProductCollection();

    } while ($pageNumber <= $pages);
}

I added the line $productCollection = $this->_getProductCollection(); after clearing the original product collection, and now pagination works as expected. I'm still unsure as to why the original code does not work.

Licensed under: CC-BY-SA with attribution
Not affiliated with magento.stackexchange
scroll top