Magento 2.1.6 Product Grid page count and record count issue when programmatically adding filter using different methods

Question

I have tried several different methods to artificially add an extra filter to the product grid (Category page).

There is an issue where after adding a filter to the product collection, the total record count is not updated, the pagination shows a wrong number of pages and the layered navigation filters does not re-calculate the product count per filter.

EDIT: Just to make it clear, all the methods I tried did filter the products correctly, but did not update the total record count (top of grid), page count (bottom of grid) and count per filter on the left nav.

Here are the methods I have tried:

Method 1

Plugin on \Magento\Catalog\Model\ResourceModel\Product\Collection

<type name="Magento\Catalog\Model\ResourceModel\Product\Collection">
    <plugin name="AddFilter" type="Company\RegionManager\Model\AddFilter"/>
</type>

with following code:

public function aroundAddFieldToFilter(ProductCollection $collection, \Closure $proceed, $fields, $condition = null)
{
    $collection->addAttributeToFilter('deal_active', array('eq' => 'yes'));

    return $fields ? $proceed($fields, $condition) : $collection;
}

Method 2

Plugin on \Magento\Catalog\Model\Layer

<type name="Magento\Catalog\Model\Layer">
    <plugin name="pluginRegionFilter" type="Company\RegionManager\Model\RegionFilter" sortOrder="1" disabled="false"/>
</type>

with following code:

public function afterGetProductCollection($subject, \Magento\Catalog\Model\ResourceModel\Product\Collection $collection) {
    $collection->addAttributeToSelect('*');
    $collection->addAttributeToFilter('deal_active', array('eq' => 'yes'));

    return $collection;
}

Method 3

Trying to get rid of caching on getSize by creating a plugin on \Magento\Framework\Data\Collection\AbstractDb

<type name="Magento\Framework\Data\Collection\AbstractDb">
    <plugin name="pluginCollectionSize" type="Company\RegionManager\Model\CollectionSize" sortOrder="1" disabled="false"/>
</type>

Using following code:

public function afterGetSize($subject, $result) {
    $sql = $subject->getSelectCountSql();
    $addSql = "SELECT count(*) FROM ($sql) AS taro01";
    $addTotal = $subject->getConnection()->fetchOne($addSql, []);
    $result += intval($addTotal) - 1;

    return $result;
}

This method combined with Method 2 gave a better result but still far, pagination is off and record count considers current page records only, not total records.

Method 4

Overriding \Magento\Catalog\Block\Product\ProductList\Toolbar

<preference for="Magento\Catalog\Block\Product\ProductList\Toolbar" type="Company\RegionManager\Block\Html\Toolbar" />

using code:

public function getTotalNum()
{
    return $this->getCollection()->getSize();
}

In the hopes of bypassing caching. (did not change anything)

Method 5

Trying to go deeper in the code.. Overriding \Magento\Catalog\Block\Product\ListProduct

<preference for="Magento\Catalog\Block\Product\ListProduct" type="Company\RegionManager\Block\Product\ListProduct" />

Overriding the _getProductCollection() method by adding a filter:

protected function _getProductCollection()
{
    if ($this->_productCollection === null) {
        $layer = $this->getLayer();
        /* @var $layer \Magento\Catalog\Model\Layer */
        if ($this->getShowRootCategory()) {
            $this->setCategoryId($this->_storeManager->getStore()->getRootCategoryId());
        }

        // if this is a product view page
        if ($this->_coreRegistry->registry('product')) {
            // get collection of categories this product is associated with
            $categories = $this->_coreRegistry->registry('product')
                ->getCategoryCollection()->setPage(1, 1)
                ->load();
            // if the product is associated with any category
            if ($categories->count()) {
                // show products from this category
                $this->setCategoryId(current($categories->getIterator()));
            }
        }

        $origCategory = null;
        if ($this->getCategoryId()) {
            try {
                $category = $this->categoryRepository->get($this->getCategoryId());
            } catch (NoSuchEntityException $e) {
                $category = null;
            }

            if ($category) {
                $origCategory = $layer->getCurrentCategory();
                $layer->setCurrentCategory($category);
            }
        }
        $this->_productCollection = $layer->getProductCollection();

        $this->prepareSortableFieldsByCategory($layer->getCurrentCategory());

        if ($origCategory) {
            $layer->setCurrentCategory($origCategory);
        }
    }
    $this->_productCollection->addAttributeToSelect('*');
    $this->_productCollection->addAttributeToFilter('deal_active', array('eq' => 'yes'));

    return $this->_productCollection;
}

Same result as all the other methods.

I am out of ideas.. anyone managed to add a filter to a category page and have the pagination, record count and layered navigation recalculate counts successfully?

Thanks!

Answer 1

I have found a method that works. It seems that the Category view, regardless of a filter being applied or not, will ALWAYS use the Catalog Search module. Knowing that, we can override the following class to add a custom filter.

Magento\CatalogSearch\Model\ResourceModel\Fulltext\Collection

But it's not that simple, first we have to create a custom filter in an xml document. Here are the steps:

Step 1

Create a search_request.xml file in Company/RegionManager/etc/search_request.xml. For my example I used the following code, you can replace any instance of deal_active with your custom attribute. Including the $deal_active$

<?xml version="1.0"?>

<requests xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:noNamespaceSchemaLocation="urn:magento:framework:Search/etc/search_request.xsd">

    <request query="catalog_view_container" index="catalogsearch_fulltext">
        <queries>
            <query xsi:type="boolQuery" name="catalog_view_container" boost="1">
                <queryReference clause="should" ref="active_query"/>
            </query>
            <query name="active_query" xsi:type="filteredQuery">
                <filterReference clause="must" ref="active_query_filter"/>
            </query>
        </queries>
        <filters>
            <filter xsi:type="termFilter" name="active_query_filter" field="deal_active" value="$deal_active$"/>
        </filters>

        <from>0</from>
        <size>10000</size>
    </request>
</requests>

Step 2

Override \Magento\CatalogSearch\Model\ResourceModel\Fulltext\Collection in the di.xml in your /etc/

<?xml version="1.0"?>

<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:ObjectManager/etc/config.xsd">
    <preference for="Magento\CatalogSearch\Model\ResourceModel\Fulltext\Collection" type="Company\RegionManager\Model\ResourceModel\Fulltext\Collection" />
</config>

Step 3

In the Company\RegionManager\Model\ResourceModel\Fulltext\Collection.php file, replace the code between the two ADDING CUSTOM FILTER comments with your custom attribute.

<?php

namespace Company\RegionManager\Model\ResourceModel\Fulltext;

use Magento\CatalogSearch\Model\Search\RequestGenerator;
use Magento\Framework\DB\Select;
use Magento\Framework\Exception\StateException;
use Magento\Framework\Search\Adapter\Mysql\TemporaryStorage;
use Magento\Framework\Search\Response\QueryResponse;
use Magento\Framework\Search\Request\EmptyRequestDataException;
use Magento\Framework\Search\Request\NonExistingRequestNameException;
use Magento\Framework\Api\Search\SearchResultFactory;
use Magento\Framework\Exception\LocalizedException;
use Magento\Framework\App\ObjectManager;

/**
 * Fulltext Collection
 * @SuppressWarnings(PHPMD.CouplingBetweenObjects)
 */
class Collection extends \Magento\CatalogSearch\Model\ResourceModel\Fulltext\Collection
{

    /**
     * @var  QueryResponse
     * @deprecated
     */
    protected $queryResponse;

    /**
     * Catalog search data
     *
     * @var \Magento\Search\Model\QueryFactory
     * @deprecated
     */
    protected $queryFactory = null;

    /**
     * @var \Magento\Framework\Search\Request\Builder
     * @deprecated
     */
    private $requestBuilder;

    /**
     * @var \Magento\Search\Model\SearchEngine
     * @deprecated
     */
    private $searchEngine;

    /**
     * @var string
     */
    private $queryText;

    /**
     * @var string|null
     */
    private $order = null;

    /**
     * @var string
     */
    private $searchRequestName;

    /**
     * @var \Magento\Framework\Search\Adapter\Mysql\TemporaryStorageFactory
     */
    private $temporaryStorageFactory;

    /**
     * @var \Magento\Search\Api\SearchInterface
     */
    private $search;

    /**
     * @var \Magento\Framework\Api\Search\SearchCriteriaBuilder
     */
    private $searchCriteriaBuilder;

    /**
     * @var \Magento\Framework\Api\Search\SearchResultInterface
     */
    private $searchResult;

    /**
     * @var SearchResultFactory
     */
    private $searchResultFactory;

    /**
     * @var \Magento\Framework\Api\FilterBuilder
     */
    private $filterBuilder;

    /**
     * @param \Magento\Framework\Data\Collection\EntityFactory $entityFactory
     * @param \Psr\Log\LoggerInterface $logger
     * @param \Magento\Framework\Data\Collection\Db\FetchStrategyInterface $fetchStrategy
     * @param \Magento\Framework\Event\ManagerInterface $eventManager
     * @param \Magento\Eav\Model\Config $eavConfig
     * @param \Magento\Framework\App\ResourceConnection $resource
     * @param \Magento\Eav\Model\EntityFactory $eavEntityFactory
     * @param \Magento\Catalog\Model\ResourceModel\Helper $resourceHelper
     * @param \Magento\Framework\Validator\UniversalFactory $universalFactory
     * @param \Magento\Store\Model\StoreManagerInterface $storeManager
     * @param \Magento\Framework\Module\Manager $moduleManager
     * @param \Magento\Catalog\Model\Indexer\Product\Flat\State $catalogProductFlatState
     * @param \Magento\Framework\App\Config\ScopeConfigInterface $scopeConfig
     * @param \Magento\Catalog\Model\Product\OptionFactory $productOptionFactory
     * @param \Magento\Catalog\Model\ResourceModel\Url $catalogUrl
     * @param \Magento\Framework\Stdlib\DateTime\TimezoneInterface $localeDate
     * @param \Magento\Customer\Model\Session $customerSession
     * @param \Magento\Framework\Stdlib\DateTime $dateTime
     * @param \Magento\Customer\Api\GroupManagementInterface $groupManagement
     * @param \Magento\Search\Model\QueryFactory $catalogSearchData
     * @param \Magento\Framework\Search\Request\Builder $requestBuilder
     * @param \Magento\Search\Model\SearchEngine $searchEngine
     * @param \Magento\Framework\Search\Adapter\Mysql\TemporaryStorageFactory $temporaryStorageFactory
     * @param \Magento\Framework\DB\Adapter\AdapterInterface $connection
     * @param string $searchRequestName
     * @param SearchResultFactory $searchResultFactory
     * @SuppressWarnings(PHPMD.ExcessiveParameterList)
     */
    public function __construct(
        \Magento\Framework\Data\Collection\EntityFactory $entityFactory,
        \Psr\Log\LoggerInterface $logger,
        \Magento\Framework\Data\Collection\Db\FetchStrategyInterface $fetchStrategy,
        \Magento\Framework\Event\ManagerInterface $eventManager,
        \Magento\Eav\Model\Config $eavConfig,
        \Magento\Framework\App\ResourceConnection $resource,
        \Magento\Eav\Model\EntityFactory $eavEntityFactory,
        \Magento\Catalog\Model\ResourceModel\Helper $resourceHelper,
        \Magento\Framework\Validator\UniversalFactory $universalFactory,
        \Magento\Store\Model\StoreManagerInterface $storeManager,
        \Magento\Framework\Module\Manager $moduleManager,
        \Magento\Catalog\Model\Indexer\Product\Flat\State $catalogProductFlatState,
        \Magento\Framework\App\Config\ScopeConfigInterface $scopeConfig,
        \Magento\Catalog\Model\Product\OptionFactory $productOptionFactory,
        \Magento\Catalog\Model\ResourceModel\Url $catalogUrl,
        \Magento\Framework\Stdlib\DateTime\TimezoneInterface $localeDate,
        \Magento\Customer\Model\Session $customerSession,
        \Magento\Framework\Stdlib\DateTime $dateTime,
        \Magento\Customer\Api\GroupManagementInterface $groupManagement,
        \Magento\Search\Model\QueryFactory $catalogSearchData,
        \Magento\Framework\Search\Request\Builder $requestBuilder,
        \Magento\Search\Model\SearchEngine $searchEngine,
        \Magento\Framework\Search\Adapter\Mysql\TemporaryStorageFactory $temporaryStorageFactory,
        \Magento\Framework\DB\Adapter\AdapterInterface $connection = null,
        $searchRequestName = 'catalog_view_container',
        SearchResultFactory $searchResultFactory = null
    ) {
        $this->queryFactory = $catalogSearchData;
        if ($searchResultFactory === null) {
            $this->searchResultFactory = \Magento\Framework\App\ObjectManager::getInstance()
                ->get('Magento\Framework\Api\Search\SearchResultFactory');
        }
        parent::__construct(
            $entityFactory,
            $logger,
            $fetchStrategy,
            $eventManager,
            $eavConfig,
            $resource,
            $eavEntityFactory,
            $resourceHelper,
            $universalFactory,
            $storeManager,
            $moduleManager,
            $catalogProductFlatState,
            $scopeConfig,
            $productOptionFactory,
            $catalogUrl,
            $localeDate,
            $customerSession,
            $dateTime,
            $groupManagement,
            $catalogSearchData,
            $requestBuilder,
            $searchEngine,
            $temporaryStorageFactory,
            $connection,
            $searchRequestName,
            $searchResultFactory
        );
        $this->requestBuilder = $requestBuilder;
        $this->searchEngine = $searchEngine;
        $this->temporaryStorageFactory = $temporaryStorageFactory;
        $this->searchRequestName = $searchRequestName;
    }

    /**
     * @inheritdoc
     */
    protected function _renderFiltersBefore()
    {
        $this->getSearchCriteriaBuilder();
        $this->getFilterBuilder();
        $this->getSearch();

        if ($this->queryText) {
            $this->filterBuilder->setField('search_term');
            $this->filterBuilder->setValue($this->queryText);
            $this->searchCriteriaBuilder->addFilter($this->filterBuilder->create());
        }

        $priceRangeCalculation = $this->_scopeConfig->getValue(
            \Magento\Catalog\Model\Layer\Filter\Dynamic\AlgorithmFactory::XML_PATH_RANGE_CALCULATION,
            \Magento\Store\Model\ScopeInterface::SCOPE_STORE
        );
        if ($priceRangeCalculation) {
            $this->filterBuilder->setField('price_dynamic_algorithm');
            $this->filterBuilder->setValue($priceRangeCalculation);
            $this->searchCriteriaBuilder->addFilter($this->filterBuilder->create());
        }

        // ADDING CUSTOM FILTER START

        $this->filterBuilder->setField('deal_active');
        $this->filterBuilder->setValue('yes');
        $this->filterBuilder->setConditionType('eq');

        $this->searchCriteriaBuilder->addFilter($this->filterBuilder->create());

        // ADDING CUSTOM FILTER END

        $searchCriteria = $this->searchCriteriaBuilder->create();
        $searchCriteria->setRequestName($this->searchRequestName);
        try {
            $this->searchResult = $this->getSearch()->search($searchCriteria);
        } catch (EmptyRequestDataException $e) {
            /** @var \Magento\Framework\Api\Search\SearchResultInterface $searchResult */
            $this->searchResult = $this->searchResultFactory->create()->setItems([]);
        } catch (NonExistingRequestNameException $e) {
            $this->_logger->error($e->getMessage());
            throw new LocalizedException(__('Sorry, something went wrong. You can find out more in the error log.'));
        }

        $temporaryStorage = $this->temporaryStorageFactory->create();
        $table = $temporaryStorage->storeApiDocuments($this->searchResult->getItems());

        $this->getSelect()->joinInner(
            [
                'search_result' => $table->getName(),
            ],
            'e.entity_id = search_result.' . TemporaryStorage::FIELD_ENTITY_ID,
            []
        );

        $this->_totalRecords = $this->searchResult->getTotalCount();

        if ($this->order && 'relevance' === $this->order['field']) {
            $this->getSelect()->order('search_result.'. TemporaryStorage::FIELD_SCORE . ' ' . $this->order['dir']);
        }
        //return parent::_renderFiltersBefore();
        return \Magento\Catalog\Model\ResourceModel\Product\Collection::_renderFiltersBefore();
    }

    /**
     * @deprecated
     * @return \Magento\Framework\Api\Search\SearchCriteriaBuilder
     */
    private function getSearchCriteriaBuilder()
    {
        if ($this->searchCriteriaBuilder === null) {
            $this->searchCriteriaBuilder = ObjectManager::getInstance()
                ->get('\Magento\Framework\Api\Search\SearchCriteriaBuilder');
        }
        return $this->searchCriteriaBuilder;
    }

    /**
     * @deprecated
     * @return \Magento\Framework\Api\FilterBuilder
     */
    private function getFilterBuilder()
    {
        if ($this->filterBuilder === null) {
            $this->filterBuilder = ObjectManager::getInstance()->get('\Magento\Framework\Api\FilterBuilder');
        }
        return $this->filterBuilder;
    }

    /**
     * @deprecated
     * @return \Magento\Search\Api\SearchInterface
     */
    private function getSearch()
    {
        if ($this->search === null) {
            $this->search = ObjectManager::getInstance()->get('\Magento\Search\Api\SearchInterface');
        }
        return $this->search;
    }

    /**
     * Return field faceted data from faceted search result
     *
     * @param string $field
     * @return array
     * @throws StateException
     */
    public function getFacetedData($field)
    {
        $this->_renderFilters();
        $result = [];
        $aggregations = $this->searchResult->getAggregations();
        // This behavior is for case with empty object when we got EmptyRequestDataException
        if (null !== $aggregations) {
            $bucket = $aggregations->getBucket($field . RequestGenerator::BUCKET_SUFFIX);
            if ($bucket) {
                foreach ($bucket->getValues() as $value) {
                    $metrics = $value->getMetrics();
                    $result[$metrics['value']] = $metrics;
                }
            } else {
                throw new StateException(__('Bucket does not exist'));
            }
        }
        return $result;
    }
}

Answer 2

Hi I did similar customisation but in my case it was for this module: https://www.manadev.com/layered-navigation-filters-multiple-select-magento-2

below code is not a solution for your problem, but if you use and install the module maybe it could help you solving your problem.

David/FixCategoryMultiStock/etc/frontend/di.xml

<type name="Magento\Catalog\Block\Product\ListProduct">
    <plugin name="fix_category_list" type="David\FixCategoryMultiStock\Plugin\ProductListPlugin"/>
</type>

<type name="Manadev\ProductCollection\Resources\Collections\FullTextProductCollection">
    <plugin name="fix_category_list" type="David\FixCategoryMultiStock\Plugin\SearchResultListPlugin"/>
</type>

I've created a helper for general action

David/FixCategoryMultiStock/Helper/FixProductCollectionHelper.php

/**
     * @param array $products
     * @param \Magento\Framework\DB\Adapter\AdapterInterface $connection
     * @param string $storeId
     * @return array $stockData
     */
    private function _getProductStockInfo($products, $connection, $storeId)
    {
        /*
         * out_stock is used to remove from product collection
         * in_stock is used for filter layered navigation
         */
        $stockData = [];
        $stockData['in_stock'] = [];
        $stockData['out_stock'] = [];

        $whId = $this->getWarehouseId($storeId);

        if (!is_null($whId)) {
            $catalog_product_super_link = $connection->getTableName('catalog_product_super_link');
            $advancedinventory_stock = $connection->getTableName('advancedinventory_stock');

            // build configurable array
            $confList = [];
            foreach ($products as $product) {
                $typeId = $product['type_id'];
                $productId = $product['entity_id'];

                // if not configurable or simple add automatically to res['in_stock'] and skip
                if (!in_array($typeId, ['configurable'])) {
                    $stockData['in_stock'][] = $productId;
                    continue;
                }
                $confList[] = $productId;
            }

            // Get Simples associated to all Configurable
            $simpleCollection = $connection->select()
                ->from($catalog_product_super_link, 'product_id')
                ->where('parent_id IN (?)', $confList);

            // TODO: Make more test for innerJoin
            $stockSql = $connection->select()
                ->from(['s' => $advancedinventory_stock], ['stock' => 'SUM(s.quantity_in_stock)'])
                ->joinInner(['p' => $catalog_product_super_link], 's.product_id = p.product_id', ['product_id' => 'p.parent_id'])
                ->where('s.place_id = ?', $whId)
                ->where('s.product_id IN (?)', $simpleCollection)
                ->group('p.parent_id');

            $stocks = $connection->fetchAll($stockSql);
            foreach ($stocks as $stock) {
                if ($stock['stock'] > 0) $stockData['in_stock'][] = $stock['product_id'];
                else $stockData['out_stock'][] = $stock['product_id'];
            }
        }

        return $stockData;
    }

and this is my plugin

/**
 * @param FullTextProductCollection $subjet
 * @param Query $query
 * @return Query
 */
public function afterGetQuery(FullTextProductCollection $subject, $query)
{
    //Get DB connection
    $connection = $subject->getConnection();

    //Get all product from sql
    $sql = $subject->getSelect();
    $products = $connection->fetchAll($sql);

    // Get products ids with stock
    $stockData = $this->fixProductCollectionHelper->getProductStockInfo($products, $connection);
    $inStockIds = $stockData['in_stock'];

    // if there are products in stock filer
    if (count($inStockIds) > 0) {
        $subject->getSelect()->where('e.entity_id IN (?)', $inStockIds);
        $query->setProductCollection($subject);
    }

    // enable flag to don't update twice
    $this->fixProductCollectionHelper->enableUpdatedFlag();

    return $query;
}

this customization removes some productos that were not in stock (the store has a multistock module)

I didn't tried with magento's default filters, but I think Method 2 is the right place why don't you try with, method prepareProductCollection

If I have time I'll try and update my answer.

Good luck.