How to append a newline after every match using xmlint --xpath

Question 1

Hello from the year 2020!

As of v2.9.9 of libxml, this behavior has been fixed in xmllint itself.

However, if you're using anything older than that, and don't want to build libxml from source just to get the fixed xmllint, you'll need one of the other workarounds here. As of this writing, the latest CentOS 8, for example, is still using a version of libxml (2.9.7) that behaves the way the OP describes.

As I gather from this SO answer, it's theoretically possible to feed a command into the --shell option of older (<2.9.9) versions of xmllint, and this will produce each node on a separate line. However, you end up having to post-process it with sed or grep to remove the visual detritus of shell mode's (human-oriented) output. It's not ideal.

XMLStarlet, if available, offers another solution, but you do need to use xmlstarlet fo to format your HTML fragment into valid XML before using xmlstarlet sel to extract nodes:

echo '
<textarea name="command" class="setting-input fixed-width"
 rows="9">1</textarea>
<textarea name="command" class="setting-input fixed-width"
 rows="5">2</textarea>' \
  | xmlstarlet fo -H -R \
  | xmlstarlet sel -T -t -v '//textarea[@name="command"]' -n

If the Attempt to load network entity message from the second xmlstarlet invocation annoys you, just add 2>/dev/null at the very end to suppress it (at the risk of suppressing other messages printed to standard error).

The XMLStarlet options explained (see also the user's guide):

fo -H -R — format the output, expecting HTML input, and recovering as much bad input as possible
- this will add an <html> root node, making the fragment in the OP's example valid XML
sel -T -t -v //xpath -n — select nodes based on XPath //xpath
- output plain text (-T) instead of XML
- using the given template (-t) that returns the value (-v) of the node rather than the node itself (allowing you to forgo using text() in the XPath expression)
- finally, add a newline (-n)

Edit(s): Removed half-implemented xmllint --shell solution because it was just bad. Added an XMLStarlet example that actually works with the OP's data.

Question 2

Try this patch, which provides 2 options:

--xpath: same as old --xpath, with nodes separated by \n.
--xpath0: same as old --xpath, with nodes separated by \0.

Test input (a.html):

<textarea name="command" class="setting-input   fixed-width" rows="9">1</textarea><textarea name="command" class="setting-input   fixed-width" rows="5">2</textarea>

Test command 1:

# xmllint --xpath '//textarea[@name="command"]/text()' --html a.html

Test output 1:

 1
 2

Test command 2:

# xmllint --xpath0 '//textarea[@name="command"]/text()' --html a.html | xargs -0 -n1

Test output 2:

 1
 2

Question 3

I did the following, ugly trick, please feel free to provide a better solution.

Changed the HTML code by replacing </textarea> with \n</textarea> using the following command:

sed 's/\<\/textarea/\'$'\n\<\\/textarea/g' f

Question 4

Below is a wrapper script intended exactly to the purpose of newlines delimited output (for old releases of xmllint).

Create a file xmllint2.sh with the contents. Then execute chmod u+x xmllint2.sh, finally run like:

./xmllint2.sh --xpath --html '//textarea[@name="command"]/text()' 2>/dev/null

(the last part of the command is to hide the output of warnings that occurs with html)

#!/bin/bash

# wrapper script to
# - have newline delimited output on Xpath querys
# - implements --xpath on very old releases

/usr/bin/xmllint --xpath &>/dev/null
implements_xpath=$?

newlines_delimited_xmllint_version=20909
current_version=$(xmllint --version |& awk 'NR==1{print $NF;exit}')

args=( "$@" )
if [[ $@ == *--xpath* ]]; then
    # iterate over positional parameters
    for ((i=0; i<${#args}; i++)); do
        if [[ ${args[i]} == --xpath ]]; then
            xpath="${args[i+1]}"
            unset args[i+1]
            unset args[i]
            break
        fi
    done
    if [[ ($implements_xpath==0 && $current_version>=20909) || $file == - || $file == /dev/stdin || $xpath == / || $xpath == string\(* ]]
    then
        exec /usr/bin/xmllint "$@"
    else
        exec /usr/bin/xmllint "${args[@]}" --shell <<< "cat $xpath" | sed '1d;$d;s/^ ------- *$//;/^$/d'
    fi
else
    exec /usr/bin/xmllint "$@"
fi

Check latest revision: https://github.com/sputnick-dev/xmllint

Debian Buster in June 29 2020 have version 2.9.4 which is 4 years old.
Debian testing/experimental have 2.9.10, which is the fixed version.

Another way to install 2.9.10 with Debian last stable: https://serverfault.com/a/1022826/120473 (without taking the risk of crashing the apt system)

Question 5

Newlines can legitimately appear in xml data. A more robust approach would delimit xpath results by a character that is guaranteed to not occur in XML data. The Null character, U+0000 in the Universal Coded Character Set, is such a character.

Note that the code point U+0000, assigned to the null control character, is the only character encoded in Unicode and ISO/IEC 10646 that is always invalid in any XML 1.0 and 1.1 document.
– https://en.wikipedia.org/wiki/Valid_characters_in_XML

@Cyker's merge request for xmllint included the addition of an -xpath0 option that would delimit xpath results by NUL. A new feature request for this functionality was opened as well.

Hopefully, xmllint will gain this feature soon.

Another xpath command line tool, xmlstarlet, can be coaxed into achieving this goal now. xmlstarlet does not currently support output of NULs directly, but we can make it output U+FFFF, which, like NUL, is guaranteed to not occur in XML data. (I assume a UTF-8 encoding of the Universal Coded Character Set.) We then just need to translate U+FFFF to U+0000 and we'll have NUL delimited xpath results.

In the following examples, I'll use the following partial html file. It's the same data from the question, except I added newlines for testing purposes.

cat > data.html <<'EOF'
<textarea name="command" class="setting-input fixed-width" rows="9">1 
 newline</textarea>
<textarea name="command" class="setting-input fixed-width" rows="5">2 
 newline</textarea>
EOF

Here is how to use xmlstarlet and a perl one-liner to delimit the xpath results with NULs:

xmlstarlet fo -H -R data.html \
| xmlstarlet sel -t -m '//textarea[@name="command"]' -v '.' -o $'\uffff' \
| perl -C -0xFFFF -l0 -pe ''

Note: I ran the HTML through xmlstarlet fo -H -R as shown in @TheDudeAbides answer.

Now that the xpath results are delimited by NULs, we can process the results with the help of xargs -0. Example:

xmlstarlet fo -H -R data.html \
| xmlstarlet sel -t -m '//textarea[@name="command"]' -v '.' -o $'\uffff' \
| perl -C -0xFFFF -l0 -pe '' \
| xargs -0 -n 1 printf '%q\n'

Result:

'1 '$'\n'' newline'
'2 '$'\n'' newline'

or load it into a bash array:

mapfile -t -d '' a < <(
 xmlstarlet fo -H -R data.html \
 | xmlstarlet sel -t -m '//textarea[@name="command"]' -v '.' -o $'\uffff' \
 | perl -C -0xFFFF -l0 -pe ''
)

declare -p a

Result:

declare -a a=([0]=$'1 \n newline' [1]=$'2 \n newline')