How to go about adding a symbol table interface to boost::spirit::lex based lexer?

Question

To implement support for typedef you'd need to lookup the symbol table when ever the lexer identifies a identifier and return a different token. This is easily done in flex lexer. I am trying to use boost Spirit to build the parser and looked about in the examples but none of them are passing any context information between the lexer and parser. What would be the simplest way to do this in the mini c compiler tutorial example?

Answer 1

That's equally easy in Spirit.Lex. All you need is the ability to invoke code after matching a token, but before returning the token to the parser. That's lexer semantic actions:

this->self += identifier[ lex::_tokenid = lookup(lex::_val) ];

where lex::_tokenid is a placeholder referring to the token id of the current token, lex::_val refers to the matched token value (at that point most probably this is a iterator_range<> pointing to the underlying input stream), and lookup is a lazy function (i.e. function object, such as a phoenix::function) implementing the actual lookup logic.

I'll try to find some time to implement a small example to be added to Spirit demonstrating this technique.

Answer 2

To implement support for typedef you'd need to lookup the symbol table when ever the lexer identifies a identifier and return a different token.

Isn't that putting the cart before the horse? The purpose of a lexer is to take text input and turn it into a stream of simple tokens. This makes the parser easier to specify and deal with, as it doesn't have to handle low-level things like "these are the possible representations of a float" and such.

The language-based mapping of an identifier token to a symbol (ie: typedef) is not something that a lexer should be doing. That's something that happens at the parsing stage, or perhaps even later as a post-process of an abstract syntax tree.

Or, to put it another way, there is a good reason why the qi::symbols is a parser object and not a lexer one. It simply isn't the lexer's business to handle this sort of thing.

In any case, it seems to me that what you want to do is build a means to (in the parser) map an identifier token to an object that represents the type that has been typedef'd. A qi::symbols parser seems to be the way to do this kind of thing.