How to manage Magento store with 60 Million Products

Question

I have completed the product import process of 60 Million products in Magento 2. But products are not visible on the front-end, while debugging this I found that re-indexing of products is still in progress.

I have tried to optimize the re-indexing so that all the products could be visible on the front-end.

While working on this re-indexing part I have analyzed the following things and now I am confused should I go ahead with Magento or not.

1. Default re-indexing with 16 core cpu and 32gb ram is too much slow, calculated assumption for re-indexing all category is more than 2 years.
2. If anyhow I do manage the first time re-indexing by increasing the server size again there will be pain on every product insert or update because Magento do the re-indexing from starting everytime.

I am not sure how the other e-commerces use the magento for millions for products.

Thanks,

Answer 1

Congratulations - I think you hold the world record for the most products in a Magento 2 catalog.

Whilst theoretically there is no limit to catalog size there will be performance issues related to such a large catalog especially with database queries.

If there is no way to reduce your catalog size then I think you need to review your hosting and database performance to try and find out where the performance issues are occurring when the indexer is running.

Perhaps getting in touch with a good Magento 2 solutions partner would be a good idea.

Answer 2

Are you amazon or ebay ?

If you have 60 million product 32gb ram would not be enough, cpu and other resources as well.

Answer 3

Have you seen this?

https://devdocs.magento.com/guides/v2.3/extension-dev-guide/indexer-batch.html

I heard this might help, but I haven't tried it myself.