what is the use of tokens.h when I am programming a lexer?

Question

I am programming a lexer in C and I read somewhere about the header file tokens.h. Is it there? If so, what is its use?

Answer 1

tokens.h is a file generated by yacc or bison that contains a list of tokens within your grammar.

Your yacc/bison input file may contain token declarations like:

%token INTEGER
%token ID
%token STRING
%token SPACE

Running this file through yacc/bison will result in a tokens.h file that contains preprocessor definitions for these tokens:

/* Something like this... */
#define INTEGER (1)
#define ID      (2)
#define STRING  (3)

Answer 2

Probably, tokens.h is a file generated by the parser generator (Yacc/Bison) containing token definitions so you can return tokens from the lexer to the parser.

With Lex/Flex and Yacc/Bison, it works like this:

parser.y:

%token FOO
%token BAR

%%

start: FOO BAR;

%%

lexer.l:

%{
#include "tokens.h"
%}

%%

foo {return FOO;}
bar {return BAR;}

%%