Select all links from a Html table using XPath (and HtmlAgilityPack)

Question

What I am trying to achieve is to extract all links with a href attribute that starts with http://, https:// or /. These links lie within a table (tbody > tr > td etc) with a certain class. I thought I could specify just the the a element without the whole path to it but it does not seem to work. I get a NullReferenceException at the line that selects the links:

var table = doc.DocumentNode.SelectSingleNode("//table[@class='containerTable']");
if (table != null)
{
    foreach (HtmlNode item in table.SelectNodes("a[starts-with(@href, 'https://')]"))
    {
        //not working

I don't know about any recommendations or best practices when it comes to XPath. Do I create overhead when I query the document two times?

Answer 1

Use:

 //tbody/descendant::a[starts-with(@href,'https://')
                     or
                       starts-with(@href,'http://')
                     or
                       starts-with(@href,'./') 
                      ]

You will still have a problem, unless you correct your code to reflect the fact that the the XmlNode.SelectNodes() instance method has a return type of XmlNodeList, not HtmlNode.

Answer 2

The problems is that you are selecting the table and then immediately trying to select the anchors as if they were direct decedents. There are tr and td tags in the middle.

So, if you change your xpath to the following, things should work:

"tbody/tr/td/a[starts-with(@href, 'https://')]"

This will not work if your anchors are wrapped up in something else, so you could select all of the anchors in the current node set (i.e. table):

"//a[starts-with(@href, 'https://')]"

See this for more detail on xpath syntax.