How to send a mail to hundreds of thousands recipients in python fast which each of recipients is showed as receiver in mail header? [closed]

Question

I'm looking of solutions to send huge number of mails (hundreds of thousands) in Python. I have a list of recipients (in a file), and I want to send a mail to all of them. I want each of recipients was show as receiver in mail header, not bcc or cc. Well, my solutions: send a separated mail to each of them not a mail to all list. I did some works below:(I used smtplib, threading):

class SendMail(threading.Thread):
  def __init__(self, from, to, subject, message):
     self.from = from
     self.to = to
     self.subject = subject
     self.message = message 
  def run(self):
    try:
                msg = MIMEMultipart('alternative')
                msg['Subject'] = self.subject
                msg['From'] = self.from
                msg['To'] = self.to
                msg.attach(MIMEText(self.message , 'html'))
                server = smtplib.SMTP()              
                server.connect('xxxxx', 25)
                server.login('cxxxxx', 'yyyyyy')                
                server.sendmail(self.from, self.to, msg.as_string())
    except:
          pass

def sendmail():
   f = open('recipients','w')
   from = "me@mail.test"
   subject = "hello"
   message = "Hello Hello"
   for line in f.readlines():
      t = SendMail(from, line, subject, message)
      t.run()
   f.close()

It work but very slow (about 6 mails/second). So please help me to make it faster. Or suggest me another solutions to do this job. Thank you very much!

Answer 1

Fastest would be to install a Mail Transfer Agent such as Postfix on your machine and hand all your emails to it for delivery using the /usr/sbin/sendmail mailing interface. Most reasonable mail servers can accept thousands of mail messages for delivery per second and may perform some SMTP pipelining to send a message to multiple recipients on a target domain in a single connection, drastically reducing traffic overhead and improving message throughput. (This wouldn't influence how your users see your emails.)

Most mail servers can also handle temporarily down servers extraordinarily well, which is extremely important since many sites use greylisting to combat spam.

If, however, you really want to be contacting an SMTP server via a network connection in your Python, it would be a very good idea to use a thread pool of sending threads that will take addresses off a queue, create and send the email, and then return to the queue for another address to service. Your current code creates and destroys a new thread for every single mail delivery. Threads take time to create and take time to destroy, and all that overhead is time that could have been spent servicing your mails.

Furthermore, the thread pool would limit the total number of active connections. There's no point in creating 1000 separate but simultaneous connections to a single mail server. The three-way handshakes that set up each session take three times the latency to your server to establish the TCP session before you can send any SMTP traffic. So create ten threads with ten connections and reuse those connections for sending emails. (Even ten is probably overkill for this -- two or three would probably work better. Heck, one thread might be best of all, though if that connection goes down (per-connection mail limits?) you would have a period of sending nothing over the wire until that connection is re-established.)

What you've created now is very similar to the thundering herd problem -- you start hundreds or thousands of threads but might not have sufficient memory to keep them all in RAM simultaneously. You might have introduced enough swapping to drastically reduce the performance of your sending system, where a single thread of execution might fit entirely in memory and run without stalling for swapping.

Answer 2

My recommendation (and by far the easiest) is to offload this work to some third party and call their libraries within your code.

MailChimp seems to be quite popular (here is their api). They also offer a generous free plan.

SocketLabs is another option (here is their api). Google should provide you tons of others.