simple antlr template rewrite

Question

I am totally new to antlr. I am trying to use antlr to do some language translation. I think I use the right syntax, but I got exception.

The following is part of the grammar:

primary
:   parExpression
|   'this' ('.' Identifier)* identifierSuffix?
|   'super' superSuffix
|   literal
|   'new' creator
|   Identifier ('.' Identifier)* identifierSuffix?
|   primitiveType ('[' ']')* '.' 'class'
|   'void' '.' 'class'
;

I added a rewrite rule, for example

|   'new' creator -> 'mynew' creator

The exception happends:

[11:11:48] error(100): rjava_new_rewrite.g:851:26: syntax error: antlr: NoViableAltException(58@[921:1: rewrite_alternative options {k=1; } : ({...}? => rewrite_template | {...}? => ( rewrite_element )+ -> {!stream_rewrite_element.hasNext()}? ^( ALT[LT(1),"ALT"] EPSILON["epsilon"] EOA["<end-of-alt>"] ) -> ^( ALT[LT(1),"ALT"] ( rewrite_element )+ EOA["<end-of-alt>"] ) | -> ^( ALT[LT(1),"ALT"] EPSILON["epsilon"] EOA["<end-of-alt>"] ) | {...}? ETC );])
[11:11:48] error(100): rjava_new_rewrite.g:851:34: syntax error: antlr: MissingTokenException(inserted [@-1,0:0='<missing SEMI>',<52>,851:33] at creator)
[11:11:48] error(100): rjava_new_rewrite.g:852:5: syntax error: antlr: MissingTokenException(inserted [@-1,0:0='<missing COLON>',<54>,852:4] at |)
[11:11:48] error(100): rjava_new_rewrite.g:0:1: syntax error: assign.types: MismatchedTreeNodeException(0!=3)
[11:11:48] error(100): rjava_new_rewrite.g:0:1: syntax error: assign.types: MismatchedTreeNodeException(3!=28)
[11:11:48] error(100): rjava_new_rewrite.g:0:1: syntax error: assign.types: MismatchedTreeNodeException(3!=27)
[11:11:48] java.util.NoSuchElementException: can't look backwards more than one token in this stream
    at org.antlr.runtime.misc.LookaheadStream.LB(LookaheadStream.java:159)
    at org.antlr.runtime.misc.LookaheadStream.LT(LookaheadStream.java:120)
    at org.antlr.runtime.RecognitionException.extractInformationFromTreeNodeStream(RecognitionException.java:144)
    at org.antlr.runtime.RecognitionException.<init>(RecognitionException.java:111)
    at org.antlr.runtime.MismatchedTreeNodeException.<init>(MismatchedTreeNodeException.java:42)
    at org.antlr.runtime.tree.TreeParser.recoverFromMismatchedToken(TreeParser.java:135)
    at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115)
    at org.antlr.grammar.v3.AssignTokenTypesWalker.grammar_(AssignTokenTypesWalker.java:388)
    at org.antlr.tool.CompositeGrammar.assignTokenTypes(CompositeGrammar.java:337)
    at org.antlr.tool.Grammar.setGrammarContent(Grammar.java:605)
    at org.antlr.works.grammar.antlr.ANTLRGrammarEngineImpl.createNewGrammar(ANTLRGrammarEngineImpl.java:192)
    at org.antlr.works.grammar.antlr.ANTLRGrammarEngineImpl.createParserGrammar(ANTLRGrammarEngineImpl.java:225)
    at org.antlr.works.grammar.antlr.ANTLRGrammarEngineImpl.createCombinedGrammar(ANTLRGrammarEngineImpl.java:203)
    at org.antlr.works.grammar.antlr.ANTLRGrammarEngineImpl.createGrammars(ANTLRGrammarEngineImpl.java:165)
    at org.antlr.works.grammar.engine.GrammarEngineImpl.getGrammarLanguage(GrammarEngineImpl.java:115)
    at org.antlr.works.components.GrammarWindowMenu.getEditTestRigTitle(GrammarWindowMenu.java:244)
    at org.antlr.works.components.GrammarWindowMenu.menuItemState(GrammarWindowMenu.java:529)
    at org.antlr.works.components.GrammarWindow.menuItemState(GrammarWindow.java:440)
    at org.antlr.xjlib.appkit.menu.XJMainMenuBar.refreshMenuItemState(XJMainMenuBar.java:175)
    at org.antlr.xjlib.appkit.menu.XJMainMenuBar.refreshMenuState(XJMainMenuBar.java:169)
    at org.antlr.xjlib.appkit.menu.XJMainMenuBar.refreshState(XJMainMenuBar.java:153)
    at org.antlr.xjlib.appkit.menu.XJMainMenuBar.refresh(XJMainMenuBar.java:145)
    at org.antlr.works.grammar.decisiondfa.DecisionDFAEngine.refreshMenu(DecisionDFAEngine.java:203)
    at org.antlr.works.components.GrammarWindow.afterParseOperations(GrammarWindow.java:1179)
    at org.antlr.works.components.GrammarWindow.access$200(GrammarWindow.java:96)
    at org.antlr.works.components.GrammarWindow$AfterParseOperations.threadRun(GrammarWindow.java:1553)
    at org.antlr.works.ate.syntax.misc.ATEThread.run(ATEThread.java:152)
    at java.lang.Thread.run(Thread.java:680)

Can anyone give any idea?

Answer 1

qinsoon wrote:

I think I use the right syntax, but I got exception.

You thought wrong. :)

You cannot put things in a rewrite rule that were not matched by the parser. So in your case:

|   'new' creator -> 'mynew' creator

'mynew' is wrong because the parser never encountered such a token/rule. If you want to insert tokens in your AST the parser didn't encounter (they're called imaginary tokens in ANTLR), you'll need to define them in the tokens {...} of your grammar, like this:

grammar YourGrammarName;

options {
  output=AST;
}

tokens {
  MYNEW;
}

primary
:   ...
|   'new' creator -> ^(MYNEW creator)
|   ...
;

This will insert a node with the type MYNEW and innner-text "MYNEW". If you want to associate some custom text in the node, do it like this:

primary
:   ...
|   'new' creator -> ^(MYNEW["mynew"] creator)
|   ...
;

As you can see from the above, I created an AST where the root node is MYNEW. If I do:

|   'new' creator -> MYNEW creator

ANTLR will return 2 nodes from that rule. You'd get in trouble if that rule would ever become the root of another (sub) tree: after all, you can get an AST with 2 roots! Always strive to let rewrite rules produce either a single AST/node:

rule
 : subrule ';' -> subrule // omit the semi-colon
 ;

or when more nodes need to be created, make a proper AST with a single root-node:

rule
 : AToken subrule1 subrule2 ';' -> ^(AToken subrule1 subrule2) // AToken is the root
 ;