Console command becomes increasingly slow

Question

I have a CSV file with about 6K SKUs that I need to loop through to update the products in Magento 2. I'm having issue where the product saves become increasingly slow, so much so that after letting it run for 24 hours, it's still not done.

I checked memory_get_usage and memory_get_peak_usage, and it hovers around 275MB for the duration, so it doesn't seem to be memory related.

I'm saving the products using ProductRepository::save. I'm stepping through the stack, and I see saving the product attributes is slow. Dispatching the events seems to be even slower, but have no idea why the saving comes to a crawl only increasingly and not from the start.

I tried production mode. This is a vanilla installation other than the module I'm working on. Any ideas?

Answer 1

Decided to convert to an answer.

Do you have index on save enabled?

If so consider on schedule plus run index process in code at end of data process. I've personally seen huge savings.

At a guess MySQL is utilising a lot of memory moving stuff in and out of tmp tables.

Answer 2

you need to check if there any observer implemented on top of product save. many 3rd party extension has this functionality. you can check this by following the command in app folder

find app/ -name events.xml | xargs grep catalog_product_save

this will give you a list of events.xml file which has events for catalog_product_save