Best way to import 80.000 products in Magento every night?

Question

So I've been asked by a client for an online store in which he need each night to delete all the products and import again aprox. 80.000 products. Of course I thought about Magento. But as we all know the importing process is a pain in the ass.

The best option is to use MAGMI of course and the import doesn't take long and it also has a nice utility that quickly empties the database and resets the product ID counter. The problem is with the bloody reindex operation.

MAGMI has a plugin called Magento Reindexer which doesn't work in my case. I guess it needs to ensure security configuration enable "shell_exec()" calls from php. I dunno how to do that. To my only choice is to reindex using the browser from the admin. Of course it's not the best solution specially since it times out and it doesn't work.

So I need to give the client two options:

1. to create a cron job that would use MAGMI's Clear Catalog utility to empty the database and then use MAGMI to import the same CSV file every night. So all the client would have to do is reupload the CSV file in the same location every night.

2. Give the client the possibility to manually make the import when he wants using MAGMI and of course empty the database using the same Clear Catalog utility from the MAGMI interface.

Either way I have to find a solution for the reindexing problem.

Does anyone have a similar experience or maybe a solution?

Thanks in advance!

Answer 1

You do not need to empty the database with each import. Magmi has the ability to update products on the fly without deleting them first.

If your concern is removing products that are no longer in the CSV, you can use a plugin I wrote and posted on StackOverflow here: https://stackoverflow.com/a/18447266/1332068

The indexer issue has also been asked about and answered on this site as well.
https://stackoverflow.com/a/19384326/1332068

I urge you to search thoroughly before asking questions as there are several solutions already available for Magmi on this site.