Free Lexical Analyzer / Scanner Source

Question

I'm looking for a hand-written free lexer/scanner for any language, even a toy language written in a language similiar to C/C++. I am a beginner in creating languages and compiler design, I'd like to play around with the source.

For example, theres a hand written lexer on this website before the flex generated lex: http://en.wikipedia.org/wiki/Flex_lexical_analyser

Thanks.

Answer 1

I like the LLVM tutorial -- it starts by presenting a hand-written lexer for a toy language called Kaleidoscope, first as a "literate programming" exercise with good textual explanations of each bit; it continues by building a parser that builds an abstract syntax tree in the same style; then the whole thing (400 lines of hand-writted standalone C++ for lexer, parser and AST builder) is shown again as a complete .cpp source file, so you don't have to put it all together by yourself. Following chapters show code generation, JIT, optimization, ... a truly useful little tutorial indeed! But, you can stop with the lexer, if that's all you want to understand for now;-).

Answer 2

Lex and Yacc are the "classic" tools used for the purpose of building languages and compilers.

Answer 3

You probably want flex, which is probably the most widely used lexer on the planet.