Multiple matches for patterns in one token

Question

I need to count every occurence of "ab.", "ba.", ".ab" in some text, where "." is (as usual) any character except newline.

I wrote:

%%

ab. counter++;
ba. counter++;
.ab counter++;

%%

My problem is, that for string:

abab

my counter is equal to 1 and it should be 3. It only matches "ab." (aba). What should I do?

Answer 1

This is how flex works: it consumes the characters already matched. In this specific case, it matches ab. on aba, consumes these characters, and leaves only b in the stream.

You could theoretically try to put back characters into the stream using the yyless(n) macro, which returns all but the first n characters into the stream. In your case yyless(2) (or rather yyless(yyleng-1) ) in all rules would give you the expected results.

Alternatively, you could use the / lookahead operator: a/(b.) is matched if the current character is an "a", followed by a "b" and any character, without actually consuming the anything other than the "a". It's a powerful tool, though I was always taught that the / is dangerous, and should be avoided (with the necessary logic moved to the parser).

Flex isn't really designed for this, though. You'd be better off doing it manually in any programming language.