Grep and Sed Equivalent for XML Command Line Processing

StackOverflow https://stackoverflow.com/questions/91791

  •  01-07-2019
  •  | 
  •  

Question

When doing shell scripting, typically data will be in files of single line records like csv. It's really simple to handle this data with grep and sed. But I have to deal with XML often, so I'd really like a way to script access to that XML data via the command line. What are the best tools?

Was it helpful?

Solution

I've found xmlstarlet to be pretty good at this sort of thing.

http://xmlstar.sourceforge.net/

Should be available in most distro repositories, too. An introductory tutorial is here:

http://www.ibm.com/developerworks/library/x-starlet.html

OTHER TIPS

Some promising tools:

  • nokogiri: parsing HTML/XML DOMs in ruby using XPath & CSS selectors

  • hpricot: deprecated

  • fxgrep: Uses its own XPath-like syntax to query documents. Written in SML, so installation may be difficult.

  • LT XML: XML toolkit derived from SGML tools, including sggrep, sgsort, xmlnorm and others. Uses its own query syntax. The documentation is very formal. Written in C. LT XML 2 claims support of XPath, XInclude and other W3C standards.

  • xmlgrep2: simple and powerful searching with XPath. Written in Perl using XML::LibXML and libxml2.

  • XQSharp: Supports XQuery, the extension to XPath. Written for the .NET Framework.

  • xml-coreutils: Laird Breyer's toolkit equivalent to GNU coreutils. Discussed in an interesting essay on what the ideal toolkit should include.

  • xmldiff: Simple tool for comparing two xml files.

  • xmltk: doesn't seem to have package in debian, ubuntu, fedora, or macports, hasn't had a release since 2007, and uses non-portable build automation.

xml-coreutils seems the best documented and most UNIX-oriented.

There is also xml2 and 2xml pair. It will allow usual string editing tools to process XML.

Example. q.xml:

<?xml version="1.0"?>
<foo>
    text
    more text
    <textnode>ddd</textnode><textnode a="bv">dsss</textnode>
    <![CDATA[ asfdasdsa <foo> sdfsdfdsf <bar> ]]>
</foo>

xml2 < q.xml

/foo=
/foo=   text
/foo=   more text
/foo=   
/foo/textnode=ddd
/foo/textnode
/foo/textnode/@a=bv
/foo/textnode=dsss
/foo=
/foo=    asfdasdsa <foo> sdfsdfdsf <bar> 
/foo=

xml2 < q.xml | grep textnode | sed 's!/foo!/bar/baz!' | 2xml

<bar><baz><textnode>ddd</textnode><textnode a="bv">dsss</textnode></baz></bar>

P.S. There are also html2 / 2html.

To Joseph Holsten's excellent list, I add the xpath command-line script which comes with Perl library XML::XPath. A great way to extract information from XML files:

 xpath -q -e '/entry[@xml:lang="fr"]' *xml

You can use xmllint:

xmllint --xpath //title books.xml

Should be bundled with most distros, and is also bundled with Cygwin.

$ xmllint --version
xmllint: using libxml version 20900

See:

$ xmllint
Usage : xmllint [options] XMLfiles ...
        Parse the XML files and output the result of the parsing
        --version : display the version of the XML library used
        --debug : dump a debug tree of the in-memory document
        ...
        --schematron schema : do validation against a schematron
        --sax1: use the old SAX1 interfaces for processing
        --sax: do not build a tree but work just at the SAX level
        --oldxml10: use XML-1.0 parsing rules before the 5th edition
        --xpath expr: evaluate the XPath expression, inply --noout

There're also xmlsed & xmlgrep of the NetBSD xmltools!

http://blog.huoc.org/xmltools-not-dead.html

If you're looking for a solution on Windows, Powershell has built-in functionality for reading and writing XML.

test.xml:

<root>
  <one>I like applesauce</one>
  <two>You sure bet I do!</two>
</root>

Powershell script:

# load XML file into local variable and cast as XML type.
$doc = [xml](Get-Content ./test.xml)

$doc.root.one                                   #echoes "I like applesauce"
$doc.root.one = "Who doesn't like applesauce?"  #replace inner text of <one> node

# create new node...
$newNode = $doc.CreateElement("three")
$newNode.set_InnerText("And don't you forget it!")

# ...and position it in the hierarchy
$doc.root.AppendChild($newNode)

# write results to disk
$doc.save("./testNew.xml")

testNew.xml:

<root>
  <one>Who likes applesauce?</one>
  <two>You sure bet I do!</two>
  <three>And don't you forget it!</three>
</root>

Source: https://serverfault.com/questions/26976/update-xml-from-the-command-line-windows

Depends on exactly what you want to do.

XSLT may be the way to go, but there is a learning curve. Try xsltproc and note that you can hand in parameters.

There's also saxon-lint from command line with the ability to use XPath 3.0/XQuery 3.0. (Other command-line tools use XPath 1.0).

EXAMPLES :

http/html:

$ saxon-lint --html --xpath 'count(//a)' http://stackoverflow.com/q/91791
328

xml :

$ saxon-lint --xpath '//a[@class="x"]' file.xml

XQuery might be a good solution. It is (relatively) easy to learn and is a W3C standard.

I would recommend XQSharp for a command line processor.

I first used xmlstarlet and still using it. When the query gets tough, i need XML's xpath2 and xquery feature support I turn to xidel http://www.videlibri.de/xidel.html

JEdit has a plugin called "XQuery" which provides querying functionality for XML documents.

Not quite the command line, but it works!

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top