Why Use Lexical Analyzers?

Question

I'm building my own language using Flex, but I want to know some things:

• Why should I use lexical analyzers?
• Are they going to help me in something?
• Are they obligatory?

Answer 1

Lexical analysis helps simplify parsing because the lexemes can be treated as abstract entities rather than concrete character sequences.

You'll need more than flex to build your language, though: Lexical analysis is just the first step.

Answer 2

Any time you are converting an input string into space-separated strings and/or numeric values, you are performing lexical analysis. Writing a cascading series of else if (strcmp (..)==0) ... statements counts as lexical analysis. Even such nasty tools as sscanf and strtok are lexical analysis tools.

You'd want to use a tool like flex instead of one of the above for one of several reasons:

• The error handling can be made much better.
• You can be much more flexible in what different things you recognize with flex. For instance, it is tough to parse a C-format hexidecimal value properly with scanf routines. scanf pretty much has to know the hex value is comming. Lex can figure it out for you.
• Lex scanners are faster. If you are parsing a lot of files, and/or large ones, this could become important.

Answer 3

You would consider using a lexical analyzer because you could use BNF (or EBNF) to describe your language (the grammar) declaratively, and then just use a parser to parse a program written in your language and get it in a structure in memory and then manipulate it freely.

It's not obligatory and you can of course write your own, but that depends on how complex the language is and how much time you have to reinvent the wheel.

Also, the fact that you can use a language (BNF) to describe your language without changing the lexical analyzer itself, enables you to make many experiments and change the grammar of your language until you have exactly what it works for you.