speed up parsing in html agility pack
-
03-06-2021 - |
Question
This is a method I use to grab certain tags with the html agility pack. I use this method to do rankings with google local. It seems to take quite a bit of time and be memory intensive, does anyone have any suggestions to make it better?
private void findGoogleLocal(HtmlNode node) {
String name = String.Empty;
//
// ----------------------------------------
if (node.Attributes["id"] != null) {
if (node.Attributes["id"].Value.ToString().Contains("panel_") && node.Attributes["id"].Value.ToString() != "panel__")
{
GoogleLocalResults.Add(new Result(URLGoogleLocal, Listing, node, SearchEngine.Google, SearchType.Local, ResultType.GooglePlaces));
}
}
if (node.HasChildNodes) {
foreach (HtmlNode children in node.ChildNodes) {
findGoogleLocal(children);
}
}
}
Solution
Why does this method have to be recursive? Just get all the nodes in one go (example using the Linq support in HAP):
var results = node.Descendants()
.Where(x=> x.Attributes["id"]!= null &&
x.Attributes["id"].Value.Contains("panel_") &&
x.Attributes["id"].Value!= "panel__")
.Select( x=> new Result(URLGoogleLocal, Listing, x, SearchEngine.Google, SearchType.Local, ResultType.GooglePlaces));
OTHER TIPS
I just want to add another clean, simple and fast solution: using XPath.
var results = node
.SelectNodes(@"//*[contains(@id, 'panel_') and @id != 'panel__']")
.Select(x => new Result(URLGoogleLocal, Listing, x, SearchEngine.Google, SearchType.Local, ResultType.GooglePlaces));
foreach (var result in results)
GoogleLocalResults.Add(result);
Fizzler: CSS selector engine for HAP
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow