Two word token using PLY in Python

Question

I am writing a compiler as part of a lab excercise and have chosen to do it in Python using PLY. I have spent some time trying to work this particular problem out and have reached a dead end as have my lab helpers.

In the language I have to write, declarators are specified with two words "was a". For example:

x was a number and x became 5.

is equal to

int x; x = 5;

When parsing with PLY, I have put 'was a' as a reserved word

reserved = {
    ...
    'was a'       : 'DECLARATOR',
    ...
}

But when I parse with the PLY lexer, it treats 'was' and 'a' as separate tokens

How can I parse was a as a token of type 'DECLARATOR' without the PLY lexer splitting it up?

If any of that is unclear let me know and I will try and answer any questions as best I can

Thanks,

Pete

Answer 1

You should never have to use two words in a token. Instead split them into two seperate tokens and ensure your language enforces that one is always followed by the other.

e.g. implement a 'was' : 'DECLARATOR_WAS' token and an 'a' : 'DECLARATOR_A' token.