Flex token order

Question

I am having problem to declare the following tokens in flex: >, <, >=, <= >>,<<, >>=, <<=.

I declared like this: (flex)

ID          [_a-zA-Z][_a-zA-Z0-9]*

">"         {
                ultimo_token = ">";
                return BT_OP;
            }

"<"         {
                ultimo_token = "<";
                return LT_OP;
            }

"<<"        {
                ultimo_token = "<<";
                return ESQ_OP;
            }
">>"        {
                ultimo_token = "<<";
                return DIR_OP;
            }                               

"<="        {
                ultimo_token = "<=";
                return LE_OP;
            }

">="        {
                ultimo_token = ">=";
                    return GE_OP;
                }
">>="       {
                ultimo_token = ">>=";
                return DIR_ATRIBUICAO;
            }           

"<<="       {
                ultimo_token = "<<=";
                return ESQ_ATRIBUICAO;
                }

{ID}+       {
                ultimo_token = "IDENTIFICADOR ";
                ultimo_token += yytext;
                yylval.sval = new string(yytext) ;
                return IDENTIFICADOR;
            }

and in bison its not working here:

expressao_relacional
    : expressao_shift { $$ = $1; }
    | expressao_relacional LE_OP expressao_shift { $$ = new NOperacaoBinaria($1, Operador::LE_OP, $3, $1->linha); }
    | expressao_relacional GE_OP expressao_shift { $$ = new NOperacaoBinaria($1, Operador::GE_OP, $3, $1->linha); }
    | expressao_relacional LT_OP expressao_shift { $$ = new NOperacaoBinaria($1, Operador::LT_OP, $3, $1->linha); }
    | expressao_relacional BT_OP expressao_shift { $$ = new NOperacaoBinaria($1, Operador::BT_OP, $3, $1->linha); }
    ;

in the code example like this one: if(a > b). It ignores > and match instead of IF '(' expression ')' where expression -> identifier rule. But there is a b identifier so it report an error...

I wrote the test scanner with the -d option and it printed this:

--(end of buffer or a NUL)
--accepting rule at line 434 ("if")
if
--accepting rule at line 274 ("(")
(
--accepting rule at line 478 ("a")
a
--accepting rule at line 484 (" >> ")
--accepting rule at line 478 ("b")
b
--accepting rule at line 280 (")")
)
--accepting rule at line 484 (" ")
--accepting rule at line 446 ("then")
then
--accepting rule at line 484 ("
")
--accepting rule at line 248 ("{")
{
--accepting rule at line 484 ("

")
--accepting rule at line 259 ("}")
}
--(end of buffer or a NUL)
--accepting rule at line 484 ("
")
--(end of buffer or a NUL)
--EOF (start condition 0)

code: if(a >>b) { }

<< is matching this rule:

[ <<EOF>>\0\t\v\r\n\f]+ ; /* eat up whitespace */

why?

...

thx rici...

Answer 1

<<EOF>> is not a character, it is a special flex rule. So you cannot put it inside [...] to make it part of a character class. When you try to do that, the letters <, >, E, O and F become part of the character class.

You hardly ever need a <<EOF>> rule, anyway and you certainly should not attempt to ignore it.

Remember that flex will use the first matching pattern in a scanner definition, when more than one pattern has the same longest match. So order matters. Unless you suppressed or ignored flex's warnings, it would have told you that your "<" and "<<" rules could never match, because they were overridden by the erroneous whitespace rules. It's always important to read and try to understand warnings (or at a minimum ask about what they mean, particularly when they seem relevant.)

From your comment about EOF matching the error pattern, I suspect that you are using an empty match to trigger an error condition. Don't do that. Your error (default) rule should be the last one in the definition, and it should match .|\n (or just . if you are certain that \n will always be matched by some rule.) That will catch any otherwise unrecognized single character, but will not be triggered by the end of file, because, as I said earlier, <<EOF>> is not a character.