Improve Speed of this function - delete product images

Question

I have a Mage CE 1.9.3+ store that hooks up to an ERP. The connector has pushed duplicate images to every product (3k+) - so we now have between 20-50 redundant images per product.

I've written something that I can call via the browser to clean up these surplus images category by category but its incredibly, incredibly slow. The user enters the category id & the function. At the moment I'm calling 'surplus' to remove these surplus images.

I'm using the media api. I suspect that the product is being saved after each image is removed & possibly re-indexed?

I'm sure this can be improved & speeded up by someone who knows their Magento better than I do.

// do an IP check
if (substr($REMOTE_ADDR != "ip")) {
    echo "You are not allowed!";
    exit();
}

include('app/Mage.php');
Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID);
error_reporting(E_ALL | E_STRICT);
Mage::setIsDeveloperMode(true);//not sure why I've set this!
ini_set('display_errors', 1);
ini_set('max_execution_time', 600);
ob_implicit_flush (1);
$mediaApi = Mage::getModel("catalog/product_attribute_media_api");
//take parameters
if (isset($_GET['function'])) {
    $method=$_GET['function'];
}
if (isset($_GET['categoryId'])) {
    $categoryID=$_GET['categoryId'];
}

$category = Mage::getModel('catalog/category')->setId($categoryID);
$attributes = Mage::getSingleton('catalog/config')->getProductAttributes();
$_products = Mage::getResourceModel('catalog/product_collection')
    ->addCategoryFilter($category)
    ->addAttributeToSelect($attributes);

$i =0;
$total = count($_products);
echo 'PRODUCT count = '.$total .' <br/>';
$position = array(0,1,2,3,4,5,6,7);
$count = 0;
$mediaApi = Mage::getModel("catalog/product_attribute_media_api");
foreach($_products as $_prod) {
    $prodID = $_prod->getId();
    $_product = Mage::getModel('catalog/product')->load($prodID);
    $items = $mediaApi->items($_product->getId());
    if (count($items) > 0) {
        foreach($items as $item) {
            if ($method == 'surplus') {
                if (in_array($item['position'], $position)) {
                } else {
                    removeSurplusImages($mediaApi, $_product, $item);
                    $count++;
                }
            } elseif ($method=='alts') {
                removeImagesWithAltTag($mediaApi,$_product, $item);
                $count++;
            }
        }
        echo "PRODUCT ". $_prod->getSku() . " done \n";
    } else {
        echo "<br>PRODUCT ". $_prod->getSku() . " has no images \n";
    }
}//EOF foreach

echo "<br/> finished removed product images $count  ";

/**
 * @param $mediaApi
 * @param $_product
 * @param $item
 * Deletes all images that have a position > 7. Checks that main or base
   image is not deleted
 * There may be > 1 image with position 6, for example. Such images will not
   be deleted
 */

function removeSurplusImages($mediaApi,$_product, $item)
{
    if ($_product->getSmallImage() != $item['file']
        && $_product->getThumbnail() != $item['file']
        && $_product->getImage() != $item['file']) {
        $mediaApi->remove($_product->getId(), $item['file']);
    }
    return;
}

/**
 * @param $mediaApi
 * @param $_product
 * @param $item
 * Deletes all images that have a label/alt tag.
 */
function removeImagesWithAltTag($mediaApi,$_product, $item)
{
    if ($item['label'] !='') {
        $mediaApi->remove($_product->getId(), $item['file']);
    }
    return;
}

Thanks for any help.

Answer 1

The main problem in your case is that you're loading products in a loop by calling:

$_product = Mage::getModel('catalog/product')->load($prodID);

in your foreach loop. This is very very bad in terms of performance. However, according to your code, you're only using the $_product variable to retrieve the image, small_image and thumbnail.

However, I'm pretty sure those 3 attributes are automatically added to select when you call:

$attributes = Mage::getSingleton('catalog/config')->getProductAttributes();

And then uses the $attributes variable to add the attribute to select.

My suggestion is to remove this line from your code:

$_product = Mage::getModel('catalog/product')->load($prodID);

And test the code that checks the image attributes.

If you see that an attribute is missing you can simply add the following line to your collection generation.

For example, if the base image is missing you can replace:

$_products = Mage::getResourceModel('catalog/product_collection')
   ->addCategoryFilter($category)
   ->addAttributeToSelect($attributes);

With:

$_products = Mage::getResourceModel('catalog/product_collection')
   ->addCategoryFilter($category)
   ->addAttributeToSelect($attributes)
   ->addAttributeToSelect('image');

Answer 2

If you wanr to remove all images with position > 7 - without further checks - this can work too:

<?php
require_once('./app/Mage.php');
error_reporting(E_ALL);
ini_set('display_errors', 1);

umask(0);
Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID);

$resource = Mage::getSingleton('core/resource');
$read     = $resource->getConnection('core_read');
$write    = $resource->getConnection('core_write');

$galleryTable       = $resource->getTableName('catalog_product_entity_media_gallery');
$galleryValueTable  = $resource->getTableName('catalog_product_entity_media_gallery_value');

$categoryId = 41;
$category   = Mage::getModel('catalog/category')->setId($categoryId);
$productIds = Mage::getResourceModel('catalog/product_collection')
    ->addCategoryFilter($category)
    ->getAllIds();

# get all image ids
$query = $read->select()
    ->from($galleryTable, array('value_id'))
    ->where('entity_id IN (?)', $productIds);
$values = $read->fetchCol($query, 'value_id');

# get all image ids w/ pos > 7
$query = $read->select()
    ->from($galleryValueTable, array('value_id'))
    ->where('position > ?', 7)
    ->where('value_id IN (?)', $values);
$values = $read->fetchCol($query, 'value_id');

$condition  = array(
    $write->quoteInto('value_id IN (?)', $values)
);

$write->delete($galleryTable, $condition);
$write->delete($galleryValueTable, $condition);

or ... for all products

# get all image ids w/ pos > 7
$query = $read->select()
    ->from($galleryValueTable, array('value_id'))
    ->where('position > ?', 7);
$values = $read->fetchCol($query, 'value_id');

$condition  = array(
    $write->quoteInto('value_id IN (?)', $values)
);

$write->delete($galleryTable, $condition);
$write->delete($galleryValueTable, $condition);

---

Edit: another bad thing for performance is this line ... it loads the product collection.

$total = count($_products);

Better use

$total = $_products->getSize();

---

Edit 2 to keep base/small image and thumbnails you can add this:

$eavAttributeTable  = $resource->getTableName('eav_attribute');
$productImageTable  = $resource->getTableName('catalog_product_entity_varchar');

After getting your $productIds add

# keep "frontend" images
# filter by frontend model should work for custom image attributes too
$query = $read->select()
    ->from($eavAttributeTable, array('attribute_id'))
    ->where('frontend_model = ?', 'catalog/product_attribute_frontend_image');
$attributeIds = $read->fetchCol($query, 'attribute_id');

$query = $read->select()
    ->distinct()
    ->from($productImageTable, array('value'))
    ->where('attribute_id IN (?)', $attributeIds)
    ->where('entity_id IN (?)', $productIds);
$keepImages = $read->fetchCol($query, 'value');

Add this to # get all image ids query

    ->where('value NOT IN (?)', $keepImages)