Python scraper (Scraperwiki) only getting half of the table

Question

I'm learning how to write scrapers using Python in Scraperwiki. So far so good, but I have spent a couple of days scratching my head now over a problem I can't get my head around. I am trying to take all links from a table. It works, but from the list of links which go from 001 to 486 it only ever starts grabbing them at 045. The url/source is just a list of cities on a website, the source can be seen here:
http://www.tripadvisor.co.uk/pages/by_city.html and the specific html starts here:

  </td></tr>
  <tr><td class=dt1><a href="by_city_001.html">'s-Gravenzande, South Holland Province&nbsp;&nbsp;&nbsp;-&nbsp;&nbsp;&nbsp;Aberystwyth, Ceredigion, Wales</a></td>
  <td class=dt1><a href="by_city_244.html">Los Corrales de Buelna,  Cantabria&nbsp;&nbsp;&nbsp;-&nbsp;&nbsp;&nbsp;Lousada, Porto District, Northern Portugal</a></td>
  </tr>
  <tr><td class=dt1><a href="by_city_002.html">Abetone, Province of Pistoia, Tuscany&nbsp;&nbsp;&nbsp;-&nbsp;&nbsp;&nbsp;Adamstown, Lancaster County, Pennsylvania</a>       /td>
  <td class=dt1><a href="by_city_245.html">Louth, Lincolnshire, England&nbsp;&nbsp;&nbsp;-&nbsp;&nbsp;&nbsp;Lucciana, Haute-Corse, Corsica</a></td>
  </tr>
  <tr><td class=dt1><a href="by_city_003.html">Adamswiller, Bas-Rhin, Alsace&nbsp;&nbsp;&nbsp;-&nbsp;&nbsp;&nbsp;Aghir, Djerba Island, Medenine Governorate</a>       </td>
  <td class=dt1><a href="by_city_246.html">Luccianna, Haute-Corse, Corsica&nbsp;&nbsp;&nbsp;-&nbsp;&nbsp;&nbsp;Lumellogno, Novara, Province of Novara, Piedmont</a></td>
  </tr>

What I am after is the links from "by_city_001.html" through to "by_city_486.html". Here is my code:

  def scrapeCityList(pageUrl):

html = scraperwiki.scrape(pageUrl)
root = lxml.html.fromstring(html)

print html

links = root.cssselect('td.dt1 a')

for link in links:

    url = 'http://www.tripadvisor.co.uk' + link.attrib['href']
    print url 

Called in the code as follows:

  scrapeCityList('http://www.tripadvisor.co.uk/pages/by_city.html')

Now when I run it, it only ever returns the links starting at 0045!

The output (045~486)

http://www.tripadvisor.co.ukby_city_045.html
http://www.tripadvisor.co.ukby_city_288.html
http://www.tripadvisor.co.ukby_city_046.html
http://www.tripadvisor.co.ukby_city_289.html
http://www.tripadvisor.co.ukby_city_047.html
http://www.tripadvisor.co.ukby_city_290.html and so on...

I've tried changing the selector to:

  links = root.cssselect('td.dt1')

And it grabs 487 'elements' like this:

<Element td at 0x13d75f0>
<Element td at 0x13d7650>
<Element td at 0x13d76b0>

But I'm not able to get the 'href' value from this. I can't figure out why it loses the first 44 links when I select 'a' in the cssselect line. I've looked at the code but I have no clue.

Thanks in advance for any help!

Claire

Answer 1

Your code works fine. You can see it in action here: https://scraperwiki.com/scrapers/tripadvisor_cities/

I've added in saving to the datastore so you can see that it actually processes all the links.

import scraperwiki
import lxml.html

    def scrapeCityList(pageUrl):
    html = scraperwiki.scrape(pageUrl)
    root = lxml.html.fromstring(html)
    links = root.cssselect('td.dt1 a')
    print len(links)
    batch = []
    for link in links[1:]: #skip the first link since it's only a link to tripadvisor and not a subpage
        record = {}
        url = 'http://www.tripadvisor.co.uk/' + link.attrib['href']
        record['url'] = url
        batch.append(record)
    scraperwiki.sqlite.save(["url"],data=batch)

scrapeCityList('http://www.tripadvisor.co.uk/pages/by_city.html') 

If you use the second css selector:

links = root.cssselect('td.dt1')

then you are selecting the td element and not the a element (which is a sub-element of the td). You could select the a by doing this:

url = 'http://www.tripadvisor.co.uk/' + link[0].attrib['href']

where you are selecting the first sub-element of the td (that's the [0]).

If you want to see all attributes of an element in lxml.html then use:

print element.attrib

which for the td gives:

{'class': 'dt1'}
{'class': 'dt1'}
{'class': 'dt1'}
...

and for the a:

{'href': 'by_city_001.html'}
{'href': 'by_city_244.html'}
{'href': 'by_city_002.html'}
...