Flex can not match ends-with symbol ($) when using yy_scan_string

Question

I want to use the flex with string buffer, not the default stdin, so I'm using yy_scan_string. It almost works fine, except the "ends-with" pattern.

e.g.

%%
ab$  {//do something}
%%

ab$ means matching "ab" if it is exact the ending string, using stdin as input, it works, but with yy_scan_string, it's not.

Test:

%option noyywrap

%{
#include <stdio.h>
%}

%%
ab$    { printf("match ab$\n"); }
%%

int main(int argc, char ** argv)
{
    if (argc > 1)
        yy_scan_string(argv[1]);

    yylex();

    if (argc > 1)
        yylex_destroy();
}

Then with flex test.l; gcc lex.yy.c; if using echo ab | ./a.out, it prints match ab$, and works fine, but if using ./a.out ab, it only prints the input ab, not match that ab$ rule.

Answer 1

I'm surprised this question has remained unanswered for so long, as its a very straightforward one, which is covered in the manual. If you look at the manual section on patterns it explains how the $ pattern works:

‘r$’ an ‘r’, but only at the end of a line (i.e., just before a newline). Equivalent to ‘r/\n’.

Note that flex's notion of “newline” is exactly whatever the C compiler used to compile flex interprets ‘\n’ as; in particular, on some DOS systems you must either filter out ‘\r’s in the input yourself, or explicitly use ‘r/\r\n’ for ‘r$’.

So you can see the manual explicitly says that the '$' does not equate to ends-with. It is not really a right string anchor as you might have found in, say, SNOBOL.

To get your program to treat command line arguments the same as the shell input, you need to add code to emulate the characters added by the operating system when performing the echo ab |. These would be \n and EOT The EOT is not matched by flex and does not actually mark end-of-file, but I was being a pedant on my emulation.

%option noyywrap

%{
#include <stdio.h>
%}

%%
ab$    { printf("match ab$\n"); }
%%

int main(int argc, char ** argv)
{   char  * buffer;
    if (argc > 1) {
        int length = strlen(argv[1]);
        buffer = (char *)malloc(length+3);
        strncpy(buffer,argv[1],length);
        buffer[length] = '\n';
        buffer[length+1] = 4; /* EOT or Cntrl-D representing EOF on unix */
        buffer[length+2] = 0;
        yy_scan_string(buffer);
       }
    yylex();

    if (argc > 1)
        yylex_destroy();
}