Modifying/Filtering yytext before calling yymore()

Question

Assume that I have the following:

pattern { /* Filter/process here then call */ yymore(); }

What I would like to do is filter/process some things in yytext before calling yymore(). yytext is never going to grow in length only possibly get shorter by the processing. This would require being able to shorten yytext. As a starting point, I tried to see if there is an easy way to e.g. delete the last character of yytext, but doing something naive like

yytext[yyleng-1] = '\0'; yyleng--;

won't work. yymore() seems to append stuff after the NIL character, which would indicate that there is some pointer inside the code generated by flex that points to the current end of yytext. Is there some idiomatic way of doing this without having to write a more complicated set of rules?

Things like yyless puts things back in the input stream which is not what I want.

Answer 1

You can't do that in the C version of 'flex', as yytext points directly into the input buffer. It isn't a copy. If you modify the input buffer you will screw things up royally. flex takes extreme care to process each character once and only once, and it doesn't do any extra buffering of its own beyond whatever the source you provide to it does. I believe it temporarily puts a NUL into the buffer while executing an action, and takes it out again when the action returns, for which purpose it must therefore have a pointer to where the NUL was.

I would move that processing into the grammar at the appropriate points, where you must already be taking a copy of yytext.