Lexer and Parser [closed]

Question

Closed. This question is opinion-based. It is not currently accepting answers.

---

Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.

Closed 8 years ago.

Improve this question

I am going to build a lexer and a parser. My problem is how do i build them. I can think of 2 implementations:

1. Lexer will convert the entire input string into an array of tokens. Then the parser will just call this array.
2. Simultaneous lexical analysis and parsing. While parsing, the parser will call the lexer for the nextToken.

Which implementation is better?

Answer 1

Both are viable. #1 is more amenable to debugging because you can save the array and see it all at once. #2 offers more performance because it is a single-pass strategy, with potentially O(1) storage complexity.

If you're not sure what to do, better to stick with #1.

Answer 2

Number 2.

Write your parser in terms of token iterators, and implement the lexing in the operator++ of the token iterator. This is the approach of Boost.Spirit. Also if your lexing is simple enough you can use Boost.Tokenizer (or wrap a bit of logic around it).