Inconsistent token handling in ANTLR4

Question

The ANTLR4 book references a multi-mode example

https://github.com/stfairy/learn-antlr4/blob/master/tpantlr2-code/lexmagic/ModeTagsLexer.g4

lexer grammar ModeTagsLexer;

// Default mode rules (the SEA)
OPEN  : '<'     -> mode(ISLAND) ;       // switch to ISLAND mode
TEXT  : ~'<'+ ;                         // clump all text together

mode ISLAND;
CLOSE : '>'     -> mode(DEFAULT_MODE) ; // back to SEA mode 
SLASH : '/' ;
ID    : [a-zA-Z]+ ;                     // match/send ID in tag to parser

https://github.com/stfairy/learn-antlr4/blob/master/tpantlr2-code/lexmagic/ModeTagsParser.g4

parser grammar ModeTagsParser;

options { tokenVocab=ModeTagsLexer; } // use tokens from ModeTagsLexer.g4

file: (tag | TEXT)* ;

tag : '<' ID '>'
    | '<' '/' ID '>'
    ;

I'm trying to build on this example, but using the « and » characters for delimiters. If I simply substitute I'm getting error 126

cannot create implicit token for string literal in non-combined grammar: '«'

In fact, this seems to occur as soon as I have the « character in the parser tag rule.

tag : '«' ID '>';

with

OPEN    : '«'   -> pushMode(ISLAND);
TEXT    : ~'«'+;

Is there some antlr foo I'm missing? This is using antlr4-maven-plugin 4.2.

The wiki mentions something along these lines, but the way I read it that's contradicting the example on github and anecdotal experience when using <. See "Redundant String Literals" at https://theantlrguy.atlassian.net/wiki/display/ANTLR4/Lexer+Rules

Answer 1

One of the following is happening:

1. You forgot to update the OPEN rule in ModeTagsLexer.g4 to use the following form:

   OPEN  : '«' -> mode(ISLAND) ;
   

2. You found a bug in ANTLR 4, which should be reported to the issue tracker.

Answer 2

Have you specified the file encoding that ANTLR should use when reading the grammar? It should be okay with European characters less than 255 but...