Setting multiple Cronjob for updating DB

Question

Currently am working on Data extraction from MLS and inserting into DB(with LAMP). Once everything is inserted, I want to set a Cronjob for checking incremental updates in the MLS every 2 hrs. So in the near future I want to complete more than 500 MLS and thereby 500 Crons will be running at a time.

Is there any challenge to setup more than 500 cron script written in PHP at a time?

Is it possible with Shell script?

Is there any alternative solution?

Thanks

Answer 1

Unless you have a big server you will likely hit some resource limit trying to update 500 MLS sources at once via cron.

Instead, you should look into a job server that maintains a queue and then define n number of workers to grab the top item from the queue and begin processing. Once a MLS is completed that worker is free to do another MLS. Based on resources/servers available you can tweak how many parallel updates can run via workers

Look into something like http://gearman.org/ or http://resquework.org/

Practically every alternative can be found here: http://queues.io/