problem sending emails from a PHP app / linux server

Question

I have a problem sending email from a linux machine. The application sends several different emails, and it uses PHP (the mail() function) / qmail (SMTP) to send emails. Some emails are sent correctly to their destination, but some the clients get some phone calls from people who haven’t received it when they should. Here is a log to prove that the email is sent. But it isn't sent ? Or is it ?

Jun  8 11:46:34 ds1111 sendmail[19207]: p58AkYSx019207: from=apache, size=2053, class=0, nrcpts=1, msgid=<201106081046.p58AkYSx019207@XXX.dedicated.XXX.co.uk>, relay=apache@localhost
Jun  8 11:46:34 ds1111 /var/qmail/bin/relaylock[19208]: /var/qmail/bin/relaylock: mail from 127.0.0.1:40345 (localhost)
Jun  8 11:46:34 ds1111 sendmail[19207]: STARTTLS=client, relay=[127.0.0.1], version=TLSv1/SSLv3, verify=FAIL, cipher=DHE-RSA-AES256-SHA, bits=256/256
Jun  8 11:46:34 ds1111 qmail-queue-handlers[19210]: Handlers Filter before-queue for qmail started ...
Jun  8 11:46:34 ds1111 qmail-queue-handlers[19210]: from=apache@XXX.dedicated.XXX.co.uk
Jun  8 11:46:34 ds1111 qmail-queue-handlers[19210]: to=recipient_email_address@googlemail.com

Jun  8 11:46:34 ds1111 qmail-queue-handlers[19210]: hook_dir = '/usr/local/psa/handlers/before-queue'
Jun  8 11:46:34 ds1111 qmail-queue-handlers[19210]: recipient[3] = 'recipient_email_address@googlemail.com'


Jun  8 11:46:34 ds1111 qmail-queue-handlers[19210]: handlers dir = '/usr/local/psa/handlers/before-queue/recipient/recipient_email_address@googlemail.com'
Jun  8 11:46:34 ds1111 qmail-queue-handlers[19210]: starter: submitter[19211] exited normally
Jun  8 11:46:34 ds1111 qmail: 1307529994.690886 new msg 49580354
Jun  8 11:46:34 ds1111 qmail: 1307529994.691052 info msg 49580354: bytes 2539 from <apache@XXX.dedicated.XXX.co.uk> qp 19211 uid 2020
Jun  8 11:46:34 ds1111 sendmail[19207]: p58AkYSx019207: to=recipient_email_address@googlemail.com, ctladdr=apache (48/48), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=32053, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (ok 1307529994 qp 19210)
Jun  8 11:46:34 ds1111 qmail: 1307529994.695376 starting delivery 100548: msg 49580354 to remote recipient_email_address@googlemail.com


Jun  8 11:46:34 ds1111 qmail: 1307529994.695467 status: local 0/10 remote 1/20
Jun  8 11:46:34 ds1111 qmail-remote-handlers[19212]: Handlers Filter before-remote for qmail started ...

Jun  8 11:46:34 ds1111 qmail-remote-handlers[19212]: from=apache@XXX.dedicated.XXX.co.uk
Jun  8 11:46:34 ds1111 qmail-remote-handlers[19212]: to=recipient_email_address@googlemail.com





Jun  8 11:46:34 ds1111 qmail-remote-handlers[19212]: hook_dir = '/usr/local/psa/handlers/before-remote'
Jun  8 11:46:34 ds1111 qmail-remote-handlers[19212]: recipient[3] = 'recipient_email_address@googlemail.com'
Jun  8 11:46:34 ds1111 qmail-remote-handlers[19212]: handlers dir = '/usr/local/psa/handlers/before-remote/recipient/recipient_email_address@googlemail.com'





Jun  8 11:46:34 ds1111 sendmail[19213]: p58AkYYN019213: from=apache, size=2053, class=0, nrcpts=1, msgid=<201106081046.p58AkYYN019213@XXX.dedicated.XXX.co.uk>, relay=apache@localhost
Jun  8 11:46:34 ds1111 /var/qmail/bin/relaylock[19214]: /var/qmail/bin/relaylock: mail from 127.0.0.1:40347 (localhost)
Jun  8 11:46:34 ds1111 sendmail[19213]: STARTTLS=client, relay=[127.0.0.1], version=TLSv1/SSLv3, verify=FAIL, cipher=DHE-RSA-AES256-SHA, bits=256/256
Jun  8 11:46:34 ds1111 qmail-queue-handlers[19216]: Handlers Filter before-queue for qmail started ...
Jun  8 11:46:34 ds1111 qmail-queue-handlers[19216]: from=apache@XXX.dedicated.XXX.co.uk
Jun  8 11:46:34 ds1111 qmail-queue-handlers[19216]: to=working_recipient_email_address@googlemail.com

Jun  8 11:46:34 ds1111 qmail-queue-handlers[19216]: hook_dir = '/usr/local/psa/handlers/before-queue'
Jun  8 11:46:34 ds1111 qmail-queue-handlers[19216]: recipient[3] = 'working_recipient_email_address@googlemail.com'
Jun  8 11:46:34 ds1111 qmail-queue-handlers[19216]: handlers dir = '/usr/local/psa/handlers/before-queue/recipient/working_recipient_email_address@googlemail.com'
Jun  8 11:46:34 ds1111 qmail-queue-handlers[19216]: starter: submitter[19217] exited normally


Jun  8 11:46:34 ds1111 qmail: 1307529994.782765 new msg 49580355
Jun  8 11:46:34 ds1111 qmail: 1307529994.782801 info msg 49580355: bytes 2539 from <apache@XXX.dedicated.XXX.co.uk> qp 19217 uid 2020
Jun  8 11:46:34 ds1111 sendmail[19213]: p58AkYYN019213: to=working_recipient_email_address@googlemail.com, ctladdr=apache (48/48), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=32053, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (ok 1307529994 qp 19216)
Jun  8 11:46:34 ds1111 qmail: 1307529994.787382 starting delivery 100549: msg 49580355 to remote working_recipient_email_address@googlemail.com
Jun  8 11:46:34 ds1111 qmail: 1307529994.787469 status: local 0/10 remote 2/20



Jun  8 11:46:34 ds1111 qmail-remote-handlers[19218]: Handlers Filter before-remote for qmail started ...

Jun  8 11:46:34 ds1111 qmail-remote-handlers[19218]: from=apache@XXX.dedicated.XXX.co.uk
Jun  8 11:46:34 ds1111 qmail-remote-handlers[19218]: to=working_recipient_email_address@googlemail.com
Jun  8 11:46:34 ds1111 qmail-remote-handlers[19218]: hook_dir = '/usr/local/psa/handlers/before-remote'
Jun  8 11:46:34 ds1111 qmail-remote-handlers[19218]: recipient[3] = 'working_recipient_email_address@googlemail.com'
Jun  8 11:46:34 ds1111 qmail-remote-handlers[19218]: handlers dir = '/usr/local/psa/handlers/before-remote/recipient/working_recipient_email_address@googlemail.com'
Jun  8 11:46:35 ds1111 qmail: 1307529995.139502 delivery 100549: success: 194.154.172.203_accepted_message./Remote_host_said:_250_ok:__Message_20026590_accepted/
Jun  8 11:46:35 ds1111 qmail: 1307529995.139597 status: local 0/10 remote 1/20
Jun  8 11:46:35 ds1111 qmail: 1307529995.139626 end msg 49580355





Jun  8 11:46:35 ds1111 qmail: 1307529995.407398 delivery 100548: success: 209.85.143.27_accepted_message./Remote_host_said:_250_2.0.0_OK_1307529909_fy2si1026718wbb.104/
Jun  8 11:46:35 ds1111 qmail: 1307529995.407551 status: local 0/10 remote 0/20
Jun  8 11:46:35 ds1111 qmail: 1307529995.407617 end msg 49580354

Answer 1

sent != received. There are several things that could cause this issue:

1. The recipient's mail server failed to receive the message.
2. The recipient's mail server marked the message as spam.
3. The recipient's mail client marked the message as spam.
4. The recipient deleted the message and forgot to tell you.

and many more. Your log does not indicate errors, so you may need to examine the problem on the receiving end. You can change the content of the messages to avoid spam filters if that turns out to be the problem, and ensure that your host is not blacklisted with any of the spam blacklisting organizations.

Answer 2

By the nature of email, once it leaves your system it's out of your hands. When it leaves your system it goes via multiple hops to the recipient's host, which then passes it to the client to read. Over that time, your message has to avoid timeouts, network issues, and at least two spam checkers.

You can ensure better deliverability by checking spam ratings, eliminating "trigger" words such as Viagra or swear words, ensure that you're mailing only to people who have opted in, and checking frequently to make sure your server is not on any blacklists. My company (which sends millions of emails a month on behalf of our clients) also does per-host mail rate limiting as many hosts will deny or severely slow mailing if you've hit them too many time in a particular period. What those rules are, nobody really knows, so you have to performance tune your mailing system.

Sending via a "trusted" host such as SocketLabs will also greatly improve your deliverability.

All that being said, if anybody tells you that they can 100% guarantee the delivery of every email, I have a bridge to sell you.....