Increasing memory usage in a shell script

Question

I'm puzzled why memory usage steadily increase over a foreach loop in a shell script I wrote for Magento to index some data. There isn't any compounding variable (arrays increasing in size, etc.) and I'm re-using the same variables in each iteration. I can't post the entire code, but here are some lines..

For each iteration of the loop, check out the below. About 0.11MB of increased usage is shown at each iteration.

$childProduct = Mage::getModel('catalog/product')->load($childId);

$searchProduct = Mage::getModel('catalog/product')->load($parentId);

$childProduct->getAttributeText($this->_mappedColorAttributeCode);

$url = Mage::getBaseUrl(Mage_Core_Model_Store::URL_TYPE_WEB) . Mage::getModel('core/url_rewrite')->loadByIdPath('product/'.$searchProduct->getId())->getRequestPath();

$imgUrl = (string)Mage::helper('catalog/image')->init($searchProduct, 'small_image')->resize(135);

$colorInfo = array() which subsequently get values assigned to it, then used to created a query, $q to run

$colorInfo is passed to saveOneRow() (shorten version)

public function saveOneRow($data) {
    // Check existence first
    $now = date("Y-m-d H:i:s");
    $data['created_at'] = $now;
    $q = "INSERT INTO {$this->_tableName}...";

    try {
        $this->_writeConnection->beginTransaction();
        $this->_writeConnection->query($q);
        $this->_writeConnection->commit();

    } catch (Exception $e){
        $this->_writeConnection->rollback();
        $detail = "[{$data['sku']}] failed to write";
        $this->_writeConnection->query("INSERT INTO {$this->_logTableName} (`key_value`,`detail`,`created_at`) VALUES ({$data['entity_id']},'$detail','$now');");
    }
}

Which of these could make the usage go up incrementally?

Answer 1

I looked into this a long, long time ago but I think the problem hasn't been fixed yet, the Observers attached to the product load leak memory.

To confirm you just would need to measure the memory usage before and after the product load lines - similar to how you have already done it to get the usage for the whole iteration.

Take a look at David's presentation at this year's Imagine conference on ERP integrations. It includes a blueprint on creating a process that keeps within its memory limitations.

Answer 2

Per philwinkle answer, the product models were the cause of steadily increasing memory usage.

I recommend using something like below instead of ...->load($id). But I still don't know why loading product models makes the memory usage go up steadily.

Mage::getModel('catalog/product')->getCollection()
    ->addAttributeToSelect('my_atts')
    ->addAttributeToFilter('entity_id', $id)
    ->getFirstItem();

Answer 3

Which PHP version are you using? Older version had a bug in the garbage collection and the memory was not freed.

You can call gc_collect_cycles manually to clean your memory at the end of the loop.