Conceptually the input stream is a sequence of regular (i.e. not special) tokens. One of these tokens is the "current token". Tokens prior to the current token may be garbage --i.e the generated code keeps no pointer to them--, tokens after the current token may not have been constructed yet. The current token is the one that is used by the parser to make most of its decisions. E.g. a production like
void A() : {} { <X> <Y> <Z> | <Y> <Z> | <Z> }
will be converted to a routine that does a switch based on the kind of the current token.
void A() {
switch( <<the kind of the current token>> ) {
case X: jj_consume_token(X) ; jj_consume_token(Y) ; jj_consume_token(Z) ; return ;
case Y: jj_consume_token(Y) ; jj_consume_token(Z) ; return ;
case Z: jj_consume_token(Z) ; return ; }
A pointer to the current token is held in field token
. The jj_consume_token
method replaces this token with the next one in the input. Basically what jj_consume_token
does is this
if( token.next == null ) token.next = <<construct the next token>> ;
token = token.next ;
if( token.kind != kind) <<throw an exception>> ;
else return token ;
The kind
parameter is used to indicate the kind of token expected.