Using xmllint and xpath with a less-than-perfect HTML document?

StackOverflow https://stackoverflow.com/questions/21479855

  •  05-10-2022
  •  | 
  •  

Question

I have an HTML page that is generated by an existing tool - I cannot change the output of this tool.

However, I want to use xmllint with the --xpath option to pick out a few specific pieces of information from the downloaded webpage. The problem is that the page starts with:

<html lang=en><head>...

And xmllint throws errors nearly immediately:

html.out:2: parser error : AttValue: " or ' expected
<html lang=en><head>
           ^

The issue certainly seems to be the missing enclosing quotation marks around the value of the lang attribute. The entire page is full of this kind of issue. (Though only sporadically.)

Nearly every browser can parse this just fine - how can I convince xmllint to do so as well? I would like to avoid having to inject an intermediate step to "fix" the file. Instead, I would like to either:

1) Find a flag, validation option, etc. that helps the parser along, or:

2) Use some other tool. (But what? xmllint is always my go-to for command line XPath commands.)

Further, using just xpath results in:

> xpath html.out '//myquery...'

not well-formed (invalid token) at line 2, column 11, ...
Was it helpful?

Solution

You can enable the HTML parser in xmllint using the --html command line option. That way, you will be able to process HTML documents.

OTHER TIPS

If does not abort the parsing, you can just hide the errors with:

2>/dev/null

Then there is Xidel, which I made just for picking some data from html pages. (although it is not perfect. I was told about two malformed documents it could not handle)

xidel  html.out -e //yourquery...

You should pre-process the HTML with a lenient parser. (That's the main difference: HTML is allowed a much more lax syntax than XML.) That is, try HTML5-Tidy and let XMLLint work on the result:

input HTML
 |
 v
Tidy
 |
 v
xmllint
 |
 v
result
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top