ANTLR v3: order token to improve performance of tree walker

Question

Is it somehow possible to specify the order of the tokens generated by ANTLR v3? My goal is to order the tokens in such a way that valid tokens in my rule "expression" follow each other in order that conditions in the tree walkers (which have about 90 different branches) can be simplified to one branch. Something like

if(LA18_0 >= ARRAY_ACCESS && LA18_0 <= VariableId){}

ANTLR assigns the values to the tokens in alphabetical order. That means, the token beginning with an "a" has the lowest value, token with "z" the highest.

An example to illustrate my problem. "abstract" (which is not a valid token in my rule "expression") has value 5 right after ARRAY_ACCESS (which has value 4) and thus the condition has to look like the following:

if(LA18_0 == ARRAY_ACCESS || (LA18_0 >= Assign && LA18_0 <= VariableId)){}

That's just an example, as I said, there are about 90 branches.

I could prefix those tokens to achieve my goal but that looks rather ugly and hampers the readability. Is there another way?

Answer 1

The only way to guarantee that ANTLR uses particular constants for particular tokens is to write your own .tokens file (you'll see them produced ANTLR when you compile grammars). You can then use the tokenVocab option to specify the name of your custom tokens file.

That said, the ordering of tokens has never proven to be a significant performance issue in my experience. The C# port of ANTLR 3 approaches the performance issue by eliminating redundant range checks that appear on some code paths.