Using lxml to extract data where all elements are not known in advance

Question

I have some sgml files that are roughly standardized. However, there can be data contained within a tag that I do not know exists before I open the file and personally read it. For example, the files have addresses and generally the addresses have a street, a city, a state, a zip and a phone. Each element of the address is indicated with a tag

 <ADDRESS>
 <STREET>One Main Street
 <CITY>Gotham City
 <ZIP>99999 0123
 <PHONE>555-123-5467
 </ADDRESS>

But, for example, I have discovered that there are tags for Country, STREET1, STREET2. I have over 200K files to process and I want know if it is possible to pull out all of the elements of the addresses without having to worry about knowing the existence of unknown tags.

What I have done so far is

h=fromstring(my_data_in_a_string)
for each in h.cssselect('mail_address'):
    each.text_content()

but what I get is problematic because I can't identify where one element ends and the next begins

One Main StreetGotham City99999 0123555-123-5467

Answer 1

To get all the tags, we iter through the document like this:

Suppose your XML structure is like this:

<ADDRESS>
 <STREET>One Main Street</STREET>
 <CITY>Gotham City</CITY>
 <ZIP>99999 0123</ZIP>
 <PHONE>555-123-5467</PHONE>
 </ADDRESS>

We parse it:

>>> from lxml import etree
>>> f = etree.parse('foo.xml')  # path to XML file
>>> root = f.getroot() # get the root element
>>> for tags in root.iter(): # iter through the root element
...     print tags.tag       # print all the tags
... 
ADDRESS
STREET
CITY
ZIP
PHONE

Now suppose your XML has extra tags as well; tags you are not aware about. Since we are iterating through the XML, the above code will return those tags as well.

<ADDRESS>
         <STREET>One Main Street</STREET>
         <STREET1>One Second Street</STREET1>
        <CITY>Gotham City</CITY>
         <ZIP>99999 0123</ZIP>
         <PHONE>555-123-5467</PHONE>         
         <COUNTRY>USA</COUNTRY>    
</ADDRESS>

The above code returns:

ADDRESS
STREET
STREET1
CITY
ZIP
PHONE
COUNTRY

Now if we want to get the text of the tags, the procedure is the same. Just print tag.text like this:

>>> for tags in root.iter():
...     print tags.text
... 

One Main Street
One Second Street
Gotham City
99999 0123
555-123-5467
USA