Parsing email headers with regular expressions in python

Question

I'm a python beginner trying to extract data from email headers. I have thousands of email messages in a single text file, and from each message I want to extract the sender's address, recipient(s) address, and the date, and write it to a single, semicolon-delimitted line in a new file.

this is ugly, but it's what I've come up with:

import re

emails = open("demo_text.txt","r") #opens the file to analyze
results = open("results.txt","w") #creates new file for search results

resultsList = []

for line in emails:
    if "From - " in line: #recgonizes the beginning of a email message and adds a linebreak
        newMessage = re.findall(r'\w\w\w\s\w\w\w.*', line)
        if newMessage:
            resultsList.append("\n")        
    if "From: " in line:
        address = re.findall(r'[\w.-]+@[\w.-]+', line)
        if address:
            resultsList.append(address)
            resultsList.append(";")
    if "To: " in line:
        if "Delivered-To:" not in line: #avoids confusion with 'Delivered-To:' tag
            address = re.findall(r'[\w.-]+@[\w.-]+', line)
            if address:
                for person in address:
                    resultsList.append(person)
                    resultsList.append(";")
    if "Date: " in line: 
            date = re.findall(r'\w\w\w\,.*', line)
            resultsList.append(date)
            resultsList.append(";")

for result in resultsList:
    results.writelines(result)


emails.close()
results.close()

and here's my 'demo_text.txt':

From - Sun Jan 06 19:08:49 2013
X-Mozilla-Status: 0001
X-Mozilla-Status2: 00000000
Delivered-To: somebody_1@hotmail.com
Received: by 10.48.48.3 with SMTP id v3cs417003nfv;
        Mon, 15 Jan 2007 10:14:19 -0800 (PST)
Received: by 10.65.211.13 with SMTP id n13mr5741660qbq.1168884841872;
        Mon, 15 Jan 2007 10:14:01 -0800 (PST)
Return-Path: <nobody@hotmail.com>
Received: from bay0-omc3-s21.bay0.hotmail.com (bay0-omc3-s21.bay0.hotmail.com [65.54.246.221])
        by mx.google.com with ESMTP id e13si6347910qbe.2007.01.15.10.13.58;
        Mon, 15 Jan 2007 10:14:01 -0800 (PST)
Received-SPF: pass (google.com: domain of nobody@hotmail.com designates 65.54.246.221 as permitted sender)
Received: from hotmail.com ([65.54.250.22]) by bay0-omc3-s21.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.2668);
         Mon, 15 Jan 2007 10:13:48 -0800
Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC;
         Mon, 15 Jan 2007 10:13:47 -0800
Message-ID: <BAY115-F12E4E575FF2272CF577605A1B50@phx.gbl>
Received: from 65.54.250.200 by by115fd.bay115.hotmail.msn.com with HTTP;
        Mon, 15 Jan 2007 18:13:43 GMT
X-Originating-IP: [200.122.47.165]
X-Originating-Email: [nobody@hotmail.com]
X-Sender: nobody@hotmail.com
From: =?iso-8859-1?B?UGF1bGEgTWFy7WEgTGlkaWEgRmxvcmVuemE=?=
 <nobody@hotmail.com>
To: somebody_1@hotmail.com, somebody_2@gmail.com, 3_nobodies@yahoo.com.ar
Bcc: 
Subject: fotos
Date: Mon, 15 Jan 2007 18:13:43 +0000
Mime-Version: 1.0
Content-Type: multipart/mixed; boundary="----=_NextPart_000_d98_1c4f_3aa9"
X-OriginalArrivalTime: 15 Jan 2007 18:13:47.0572 (UTC) FILETIME=[E68D4740:01C738D0]
Return-Path: nobody@hotmail.com

The output is:

somebody_1@hotmail.com;somebody_2@gmail.com;3_nobodies@yahoo.com.ar;Mon, 15 Jan 2007 18:13:43 +0000;

This output would be fine except there's a line break in the 'From:' field in my demo_text.txt (line 24), and so I miss 'nobody@hotmail.com'.

I'm not sure how to tell my code to skip line break and still find email address in the From: tag.

More generally, I'm sure there are many more sensible ways to go about this task. If anyone could point me in the right direction, I'd sure appreciate it.

Answer 1

Your demo text is practicallly the mbox format, which can be perfectly processed with the appropriate object in the mailbox module:

from mailbox import mbox
import re

PAT_EMAIL = re.compile(r"[0-9A-Za-z._-]+\@[0-9A-Za-z._-]+")

mymbox = mbox("demo.txt")
for email in mymbox.values():
    from_address = PAT_EMAIL.findall(email["from"])
    to_address = PAT_EMAIL.findall(email["to"])
    date = [ email["date"], ]
    print ";".join(from_address + to_address + date)

Answer 2

In order to skip newlines, you can't read it line by line. You can try loading in your file, and using your keywords (From, To, etc.) as boundaries. So when you search for 'From -', you use the rest of your keywords as boundaries so they won't be included in the portion of the list.

Also, mentioning this cause you said you're a beginner: The "Pythonic" way of naming your non-class variables is with underscores. So resultsList should be results_list.