ANTLR 4 uses unbounded lookahead with non-deterministic termination conditions for the prediction process. While the TokenStream
implementations do call TokenSource.nextToken
lazily, it is not safe to ever assume that the number of tokens consumed so far is bounded.
In other words, the actual semantics of using a parser action to change the behavior of the lexer are undefined. Different versions of ANTLR 4, or even subtle changes in the input you give it, could produce completely different results.