string recursion antlr lexer token

Question

How do I build a token in lexer that can handle recursion inside as this string:

${*anything*${*anything*}*anything*}

?

Answer 1

Yes, you can use recursion inside lexer rules.

Take the following example:

${a ${b} ${c ${ddd} c} a}

which will be parsed correctly by the following grammar:

parse
  : DollarVar
  ;

DollarVar
  : '${' (DollarVar | EscapeSequence | ~Special)+ '}'
  ;

fragment 
Special
  :  '\\' | '$' | '{' | '}'
  ;

fragment
EscapeSequence
  :  '\\' Special
  ;

as the interpreter inside ANTLRWorks shows:

alt text http://img185.imageshack.us/img185/5471/recq.png

Answer 2

ANTLR's lexers do support recursion, as @BartK adeptly points out in his post, but you will only see a single token within the parser. If you need to interpret the various pieces within that token, you'll probably want to handle it within the parser.

IMO, you'd be better off doing something in the parser:

variable: DOLLAR LBRACE id variable id RBRACE;

By doing something like the above, you'll see all the necessary pieces and can build an AST or otherwise handle accordingly.