Pushing/Poping Lexical States in JavaCC

Question

I'm trying to refactor a JavaCC DSL parser that is written using mostly only one lexical state.

My goal is to introduce a new keyword that is context sensitive to not invalidate older configurations using the older DSL.

The idea was to change lexical state and introduce the new keyword so it is only valid during a very specific context. Making backward compatible with earlier releases.

Problem: Comments already change lexical state, to change back to DEFAULT after end of comment. Changing back to DEFAULT is "hardcoded", but now I would need Comments to instead change back to last active lexical state.

From what I understand keeping states in a stack, pushing and poping them, would help me achieve this (Think this is called DPDA). Is this possible in JavaCC?

[Edit, after searching some more, this is who far I got:]

TOKEN_MGR_DECLS : {
    Stack lexicalStateStack = new Stack();
}

SKIP : {
   " "
|  "\t"
|  "\n"
|  "\r"
}

MORE :
{
  "/*" { lexicalStateStack.push(curLexState); } : IN_COMMENT
}

SPECIAL_TOKEN :
{
  <IN_COMMENT : "*/" > { SwitchTo((int)lexicalStateStack.pop()); }
}

Seems to be doing what I want, but is it correct? (Thinking of lookaheads here)

Answer 1

Absolutely. You can stack the lexical states. But, do the transitions from the token manager, not the parser.

See questions 3.17 How do I tokenize nested comments? and 3.12 Can the parser force a switch to a new lexical state? in the FAQ http://www.engr.mun.ca/~theo/JavaCC-FAQ/ .