AWS SES sends emails to multiple recipients too slowly

Question

I have created a Job Queue module that processes jobs and constructs "social-network" type emails. 2 processes consists of:

1. Building the custom emails (Views) e.g. User A and User B have commented on your post or User B and User C also likes User C's post. Each recipient gets a different email. I initially created a new Swiftmailer instance and add the message content, subject and recipient. I then added these instances to the database.

2. A cron job runs to fetch and send these emails at a later time.

While benchmarking, I realised it was sending out 2 emails per second avg. So I tried storing Swift_Message Instances in the database instead. No luck though, still takes as long.

Currently, the code

• Creates a new Swift_SmtpTransport.
• Creates a new Swift_Mailer instance.
  • Loops through the Swift_Message messages retrieved from the DB
  • Sends each email.

But it still averages about 2 emails a second. Is there any way I can improve on the process to speed up delivery? I am using Amazon SES as my SMTP transport and I know it can at least handle 5 emails a second.

So it is probably something I am doing wrong. Any thoughts appreciated.

EDIT

Please keep in mind that the messages differ for each recipient. I could try out the Swift_Decorator plugin but it will mean that I will have to change the way the views are generated. I am just looking out for other alternatives to speed the process up.

Answer 1

I am using Amazon SES as my SMTP transport

In my experience using SES, the minimum API response time I've seen has been in the half second range, while the average hovered around one second. This is not a limitation of the connection, of TCP/IP, of bandwidth available, etc, but of their processing of requests per connection. Others on the official support forums report the same. The SMTP transport isn't any faster.

The only way to send faster is to send in parallel. This is the approach they've been recommending on their official support forums.

The SES API permits multiple simultaneous connections as long as you stay under the "mails per second" quota. If you don't know your current per-second quota, check your send stats. I don't know the SMTP transport handles being over this limitation.

In order to send a higher volume of mail in parallel, we modified our queue processor to be able to run in parallel. To ensure that mails were never sent twice, we added an "PID x grabbed this" column to the table and ran a query similar to UPDATE Queue SET selected_pid = ? WHERE target_ts < NOW() AND selected_pid IS NULL LIMIT X. We'd then look for mails we can send, send them all, and then try again until we ran out of mails that we'd need to send in that period.

We also modified the code that populates the queue to ensure that there would never be more mails in the queue than we could send. We were able to do this because we send mails in batches. When you have a constant trickle of mails, chances are that you won't need to jump through that hoop. Just make sure your sending code knows how to properly respond to the various over-quota errors.

As an alternate to a cron-based queue processor, consider using Gearman to set up a set of asynchronous background workers to send mail.