Import new and update products and their attributes with a CSV file and cron job Magento2

Question

I am using Magento 2.3.3 and wants to import products using cron and csv. I have google Sheet which contains all products data and sheet's data changes continuously. So I have to run a cron every day which updates product price, qty and some attributes values.

1. I've generated csv from sheet and it is being successfully imported manually.
2. Then I tried to import csv using cron by hitting some import functions existing in module- import-export. But didn't work and everything messed up.
3. Now I'm trying to import products by directly save them in database.

Is it possible to do that? Or is their another way?

Checked this link also but didn't understand that much.

Answer 1

I think you are looking for this program example, which might be best option. There is also a note about images in comments.

I would consider, having some sort of rollback strategy along with cron module, something like (pseudo code)

take product data dump 
run cron import 
  if Import successful 
     delete dump
     log something
  else if import fail
     rollback from dump
     log error and other relevant stuff 

Answer 2

I understood your concern, that you want to add new products, update existing products price and product attribute value without mess up with old data.

I recommended you to create a patch command to import-products, In this patch include below conditions

1. Check Website code and Store view condition
2. Whether the product has existing or not
3. To update any data, include additional option in command line **productupdate==1** based on this condition you update the new values with old data

After checking this condition, you need to check like below sample code.

        $import_file = pathinfo($import_path);
        $import = $this->importFactory->create();
        $import->setData(
            array(
                'entity' => 'catalog_product',
                'behavior' => $import->getDefaultBehavior(),
                'validation_strategy' => 'validation-stop-on-errors',
            )
        );

        $read_file = $this->readFactory->create($import_file['dirname']);
        $csvSource = $this->csvSourceFactory->create(
            array(
                'file' => $import_file['basename'],
                'directory' => $read_file,
            )
        );
        $validate = $import->validateSource($csvSource);
        if (!$validate) {
            $output->writeln('<error>Unable to validate the CSV.</error>');
            exit;
        }

        $result = $import->importSource();
        if ($result) {
            $import->invalidateIndex();
        }

        if ($import->getCreatedItemsCount() > 0 && $productUpdate == 0) {
            $output->writeln("<info>Finished importing products from $import_path</info>");
        } else if ($import->getUpdatedItemsCount() > 0 && $productUpdate == 1) {
            $output->writeln("<info>Finished Updating the products from $import_path</info>");
        } else {
            $output->writeln("<error> Products are not created/updated, cross check the source file from $import_path</error>");
        }

Note: The CSV file should have all data of the products while import new value Ex: If you updating price value, then the CSV file the product should have all data and include with new price value. This will avoid missing data while doing update.

I am following this and its working for me.

Answer 3

You can create a CRON workflow and reuse the module-import-export model to import any entity.

This is an example class to define the Magento import model.

<?php


use Magento\Framework\App\Filesystem\DirectoryList;
use Magento\ImportExport\Model\Import\Adapter as ImportAdapter;
use Magento\ImportExport\Model\Import\ErrorProcessing\ProcessingErrorAggregatorInterface;

class ImportCron
{
    const ENTITY = 'catalog_product'; //catalog_product, customer, advanced_pricing...
    const BEHAVIOR = 'append'; //append, replace, delete


    protected $importModel;

    protected $fileSystem;

    /**
     * ImportCron constructor.
     * @param \Magento\ImportExport\Model\Import $importModel
     * @param \Magento\Framework\Filesystem $filesystem
     */
    public function __construct(
        \Magento\ImportExport\Model\Import $importModel,
        Magento\Framework\Filesystem $filesystem

    )
    {
        $this->importModel = $importModel;
        $this->fileSystem = $filesystem;
    }

    public function execute(){

        try{
            //TODO: Set Area scope, cron, adminhtml

            //TODO: Check entity, beavior, input file exist, etc...

            $fileName = '';
            $skipValidate = true;

            $sourceFile = $this->importModel->getWorkingDir() . self::ENTITY . '.csv';
            $importedFile = $this->importModel->getWorkingDir() . $fileName;
            if (strtolower($fileName) != self::ENTITY . '.csv') {
                copy($importedFile, $sourceFile);
            }

            $data = array(
                'entity' => '',
                'based_entity' => self::ENTITY,
                'behavior' => self::BEHAVIOR,
                $this->importModel::FIELD_NAME_VALIDATION_STRATEGY => $skipValidate ? ProcessingErrorAggregatorInterface::VALIDATION_STRATEGY_SKIP_ERRORS : ProcessingErrorAggregatorInterface::VALIDATION_STRATEGY_STOP_ON_ERROR,
                $this->importModel::FIELD_NAME_ALLOWED_ERROR_COUNT => 10,
                $this->importModel::FIELD_FIELD_MULTIPLE_VALUE_SEPARATOR => \Magento\ImportExport\Model\Import::DEFAULT_GLOBAL_MULTI_VALUE_SEPARATOR,
                $this->importModel::FIELD_FIELD_SEPARATOR => ',',
                $this->importModel::FIELD_NAME_IMG_FILE_DIR => 'pub/media/import'
            );


            $this->importModel->setData($data);

            $source = ImportAdapter::findAdapterFor(
                $sourceFile,
                $this->fileSystem->getDirectoryWrite(DirectoryList::ROOT),
                $data[$this->importModel::FIELD_FIELD_SEPARATOR]
            );

            $validationResult = $this->importModel->validateSource($source);

            if (!$this->importModel->getProcessedRowsCount()) {
                if (!$this->importModel->getErrorAggregator()->getErrorsCount()) {
                } else {
                    foreach ($this->importModel->getErrorAggregator()->getAllErrors() as $error) {
                        //echo error;
                    }
                }
            } else {
                $errorAggregator = $this->importModel->getErrorAggregator();
                if (!$validationResult) {
                    foreach ($errorAggregator->getRowsGroupedByErrorCode() as $errorMessage => $rows) {
                        //echo error;
                    }
                } else {
                    if ($this->importModel->isImportAllowed()) {
                        $this->importModel->importSource();
                        $errorAggregator = $this->importModel->getErrorAggregator();
                        if ($errorAggregator->hasToBeTerminated()) {
                            foreach ($errorAggregator->getRowsGroupedByErrorCode() as $errorMessage => $rows) {
                                //echo error;
                            }
                        } else {
                            $this->importModel->invalidateIndex();
                            foreach ($errorAggregator->getRowsGroupedByErrorCode() as $errorMessage => $rows) {
                                //echo error;
                            }
                        }

                        //TODO: Move source file to archive or some folder.

                    } else {
                        //file valid but could not import
                    }
                }
            }
        }
        catch (\Exception $e){

        }

    }

}

Regards,