Scheduling Emails Using PHP + MySQL + Cron

Question

I'm writing a web app in PHP + Laravel + MySQL.

In the system, a user can schedule emails (and other API calls) at arbitrary times (much like how you schedule posts in WordPress). I can use CRON to inspect the database every 5min or so to find emails that should be sent, send them, and update their status.

However, this is a SaaS app. So the amount of emails to be sent at a particular time can grow rapidly. I can create a "lock file" every time the CRON script runs so that only one instance of it is running at a time. The lock file will be deleted after a script finishes execution.

But with potentially large data, I would want a way to process multiple messages simultaneously, potentially using multiple "workers." Is there any existing solution manage such a queue?

Answer 1

Yes! Task/Message/Job queues are what you are looking for! They allow you to put various tasks in queues from which you can retrieve them and process them, this process can scale horizontally as each worker can pull a task once its finished with the previous one.

You should have the cron maybe every minute/two minutes that just uploads the task and what needs to be done. This will make sure the cron is very quick.

Take a look at Iron.io Here is an extract from the website which gives a nice overview of these kinds of systems:

An easy-to-use scalable task queue that gives cloud developers a simple way to offload front-end tasks, run scheduled jobs, and process tasks in the background and at scale.

Gearman is also a great solution that you can use yourself and is very simple. You can send the message in many different languages and use a different langauge to process it. Say PHP -> C etc...

The Wikipedia link will tell you everything you need to know, here is a quick excerpt:

Message queues provide an asynchronous communications protocol, meaning that the sender and receiver of the message do not need to interact with the message queue at the same time. Messages placed onto the queue are stored until the recipient retrieves them.