Lexical Analysis of Python Programming Language
-
20-09-2019 - |
Question
Does anyone know where a FLEX or LEX specification file for Python exists? For example, this is a lex specification for the ANSI C programming language: http://www.quut.com/c/ANSI-C-grammar-l-1998.html
FYI, I am trying to write code highlighting into a Cocoa application. Regex won't do it because I also want grammar parsing to fold code and recognize blocks.
Solution
Lex is typically just used for tokenizing, not full parsing. Projects that use flex/lex for tokenizing typically use yacc/bison for the actual parsing.
You may want to take a look at ANTLR, a more "modern" alternative to lexx & yacc.
The ANTLR Project has a Github repo containing many ANTLR 4 grammars including at least one for Python 3.
OTHER TIPS
grammar.txt is the official, complete Python grammar -- not directly lex
compatible, but you should be able to massage it into a suitable form.
Have you considered using one of the existing code highlighters, like Pygments?