Keyword Matching in Pyparsing: non-greedy slurping of tokens

Question

Pythonistas:

Suppose you want to parse the following string using Pyparsing:

'ABC_123_SPEED_X 123'

were ABC_123 is an identifier; SPEED_X is a parameter, and 123 is a value. I thought of the following BNF using Pyparsing:

Identifier = Word( alphanums + '_' )
Parameter = Keyword('SPEED_X') or Keyword('SPEED_Y') or Keyword('SPEED_Z')
Value = # assume I already have an expression valid for any value
Entry = Identifier + Literal('_') + Parameter + Value
tokens = Entry.parseString('ABC_123_SPEED_X 123')
#Error: pyparsing.ParseException: Expected "_" (at char 16), (line:1, col:17)

If I remove the underscore from the middle (and adjust the Entry definition accordingly) it parses correctly.

How can I make this parser be a bit lazier and wait until it matches the Keyword (as opposed to slurping the entire string as an Identifier and waiting for the _, which does not exist.

Thank you.

[Note: This is a complete rewrite of my question; I had not realized what the real problem was]

Answer 1

I based my answer off of this one, since what you're trying to do is get a non-greedy match. It seems like this is difficult to make happen in pyparsing, but not impossible with some cleverness and compromise. The following seems to work:

from pyparsing import *
Parameter = Literal('SPEED_X') | Literal('SPEED_Y') | Literal('SPEED_Z')
UndParam = Suppress('_') + Parameter
Identifier = SkipTo(UndParam)
Value = Word(nums)
Entry = Identifier + UndParam + Value

When we run this from the interactive interpreter, we can see the following:

>>> Entry.parseString('ABC_123_SPEED_X 123')
(['ABC_123', 'SPEED_X', '123'], {})

Note that this is a compromise; because I use SkipTo, the Identifier can be full of evil, disgusting characters, not just beautiful alphanums with the occasional underscore.

EDIT: Thanks to Paul McGuire, we can concoct a truly elegant solution by setting Identifier to the following:

Identifier = Combine(Word(alphanums) +
        ZeroOrMore('_' + ~Parameter + Word(alphanums)))

Let's inspect how this works. First, ignore the outer Combine; we'll get to this later. Starting with Word(alphanums) we know we'll get the 'ABC' part of the reference string, 'ABC_123_SPEED_X 123'. It's important to note that we didn't allow the "word" to contain underscores in this case. We build that separately in to the logic.

Next, we need to capture the '_123' part without also sucking in '_SPEED_X'. Let's also skip over ZeroOrMore at this point and return to it later. We start with the underscore as a Literal, but we can shortcut with just '_', which will get us the leading underscore, but not all of '_123'. Instictively, we would place another Word(alphanums) to capture the rest, but that's exactly what will get us in trouble by consuming all of the remaining '_123_SPEED_X'. Instead, we say, "So long as what follows the underscore is not the Parameter, parse that as part of my Identifier. We state that in pyparsing terms as '_' + ~Parameter + Word(alphanums). Since we assume we can have an arbitrary number of underscore + WordButNotParameter repeats, we wrap that expression a ZeroOrMore construct. (If you always expect at least underscore + WordButNotParameter following the initial, you can use OneOrMore.)

Finally, we need to wrap the initial Word and the special underscore + Word repeats together so that it's understood they are contiguous, not separated by whitespace, so we wrap the whole expression up in a Combine construct. This way 'ABC _123_SPEED_X' will raise a parse error, but 'ABC_123_SPEED_X' will parse correctly.

Note also that I had to change Keyword to Literal because the ways of the former are far too subtle and quick to anger. I do not trust Keywords, nor could I get matching with them.

Answer 2

If you are sure that the identifier never ends with an underscore, you can enforce it in the definition:

from pyparsing import *

my_string = 'ABC_123_SPEED_X 123'

Identifier = Combine(Word(alphanums) + Literal('_') + Word(alphanums))
Parameter = Literal('SPEED_X') | Literal('SPEED_Y') | Literal('SPEED_Z')
Value = Word(nums)
Entry = Identifier + Literal('_').suppress() + Parameter  + Value
tokens = Entry.parseString(my_string)

print tokens # prints: ['ABC_123', 'SPEED_X', '123']

If it's not the case but if the identifier length is fixed you can define Identifier like this:

Identifier = Word( alphanums + '_' , exact=7)

Answer 3

You can also parse the identifier and parameter as one token, and split them in a parse action:

from pyparsing import *
import re

def split_ident_and_param(tokens):
    mo = re.match(r"^(.*?_.*?)_(.*?_.*?)$", tokens[0])
    return [mo.group(1), mo.group(2)]

ident_and_param = Word(alphanums + "_").setParseAction(split_ident_and_param)
value = Word(nums)
entry = ident_and_param + value

print entry.parseString("APC_123_SPEED_X 123")

The example above assumes that the identifiers and parameters always have the format XXX_YYY (containing one single underscore).

If this is not the case, you need to adjust the split_ident_and_param() method.

Answer 4

This answers a question that you probably have also asked yourself: "What's a real-world application for reduce?):

>>> keys = ['CAT', 'DOG', 'HORSE', 'DEER', 'RHINOCEROS']
>>> p = reduce(lambda x, y: x | y, [Keyword(x) for x in keys])
>>> p
{{{{"CAT" | "DOG"} | "HORSE"} | "DEER"} | "RHINOCEROS"}

Edit:

This was a pretty good answer to the original question. I'll have to work on the new one.

Further edit:

I'm pretty sure you can't do what you're trying to do. The parser that pyparsing creates doesn't do lookahead. So if you tell it to match Word(alphanums + '_'), it's going to keep matching characters until it finds one that's not a letter, number, or underscore.