Retrieve information about pre-processor directives

Question

I've recently started using libclang to parse C files. The problem I'm having is that apparently, libclang initiates the preprocessor before generating AST. I would like to prohibit the preprocessor from running, and instead be given information that preprocessor directives are in the file...

I use the following python script (cindex.py and libclang)

import codecs
from clang.cindex import *

class SourceFile(object):
    def __init__(self, path):
        with codecs.open(path, 'r', 'utf-8') as file:
            self.file_content = file.read()

        index = Index.create()
        root_node = index.parse(path)

        for included in root_node.get_includes():
            print included.include

        self.print_declerations(root_node.cursor)

    def print_declerations(self, root, recurse=True):
        print root.kind.name, root.spelling
        if root.kind.is_declaration():
            node_def = root.get_definition()
            if node_def is not None:
                start_offset = node_def.extent.start.offset
                end_offset = node_def.extent.end.offset + 1
                print self.file_content[start_offset:end_offset], '\n'

        if recurse:
            for child in root.get_children():
                self.print_declerations(child, False)

if __name__ == '__main__':
    path = 'Sample.cpp'
    print 'Translation unit:', path
    source = SourceFile(path)

Which outputs

Translation unit: Sample.cpp
/mingw/include\stdio.h
/mingw/include\_mingw.h
/mingw/include\sys/types.h
TRANSLATION_UNIT None
TYPEDEF_DECL __builtin_va_list

STRUCT_DECL _iobuf

TYPEDEF_DECL FILE

VAR_DECL _iob
UNEXPOSED_DECL 

FUNCTION_DECL main
int main()
{
    printf(HELLO_WORLD);
    return 0;
}

For the following C-code:

#include <stdio.h>
#define HELLO_WORLD "HELLO!"

int main()
{
    printf(HELLO_WORLD);
    return 0;
}

What I would like is to get DEFINE_DECL HELLO_WORLD for my #define in the code (currently I get nothing). And of course also get similar statements for my #include's. Is this possible?

EDIT: Basically, I want to parse the file without preprocessor directives expanded.

Answer 1

A few days ago I have asked the same question on the #llvm freenode irc channel. The answer was "macroses is not part of AST, so you can't", but most probably "-fsyntax-only" option and clang plugin instead of libclang may help you.

Edited: Looks like now it is actually possible, see answer by bradtgmurray

Answer 2

If you add PARSE_DETAILED_PROCESSING_RECORD as an option to your call to index.parse() you'll get access to the preprocessor nodes.

index = clang.cindex.Index.create()                                                                         
tu = index.parse(filename, options=clang.cindex.TranslationUnit.PARSE_DETAILED_PROCESSING_RECORD)

This option maps to the following libclang C API option value. There's a comment there that includes some more context.

/**                                                                         
 * \brief Used to indicate that the parser should construct a "detailed"    
 * preprocessing record, including all macro definitions and instantiations.
 *                                                                          
 * Constructing a detailed preprocessing record requires more memory        
 * and time to parse, since the information contained in the record         
 * is usually not retained. However, it can be useful for                   
 * applications that require more detailed information about the            
 * behavior of the preprocessor.                                            
 */                                                                         
CXTranslationUnit_DetailedPreprocessingRecord = 0x01,   

Answer 3

If you're using the command line arguments as a way to invoke libclang, here is the relevant code from the libclang C API implementation:

// Do we need the detailed preprocessing record?
if (options & CXTranslationUnit_DetailedPreprocessingRecord) {
  Args->push_back("-Xclang");
  Args->push_back("-detailed-preprocessing-record");
}