Start states in Lex / Flex
-
16-09-2019 - |
Question
I'm using Flex and Bison for a parser generator, but having problems with the start states in my scanner.
I'm using exclusive rules to deal with commenting, but this grammar doesn't seem to match quoted tokens:
%x COMMENT
// { BEGIN(COMMENT); }
<COMMENT>[^\n] ;
<COMMENT>\n { BEGIN(INITIAL); }
"==" { return EQUALEQUAL; }
. ;
In this simple example the line:
// a == b
isn't matched entirely as a comment, unless I include this rule:
<COMMENT>"==" ;
How do I get round this without having to add all these tokens into my exclusive rules?
Solution
Matching C-style comments in Lex/Flex or whatever is well documented:
in the documentation, as well as various variations around the Internet.
Here is a variation on that found in the Flex documentation:
<INITIAL>{
"//" BEGIN(IN_COMMENT);
}
<IN_COMMENT>{
\n BEGIN(INITIAL);
[^\n]+ // eat comment
"/" // eat the lone /
}
OTHER TIPS
Try adding a "+" after the [^n] rule. I don't know why the exclusive state is still picking up '==' even in an exclusive state, but apparently it is. Flex will normally match the rule that matches the most text, and adding the "+" will at least make the two rules tie in length. Putting the COMMENT rule first will cause it to be used in case of a tie.
The clue is:
The problem is this 'eat comment' rule doesn't seem to match tokens with more than one character
so add a * to match zero or more non-newlines. You want Zero otherwise a empty comment will not match.
%x COMMENT
// { BEGIN(COMMENT); }
<COMMENT>[^\n]* ;
<COMMENT>\n { BEGIN(INITIAL); }
"==" { return EQUALEQUAL; }
. ;