Compiler codegenerator verification

Question

I am working on a new backend for the LLVM compiler. The backend is coming along nicely but I am arriving at a point where i want to verify if the codegenerator is operating correctly. I have got a processor simulator available which is very accurate and which I would like to use for the verification.

Using the LLVM-lit doesn't seem like a good option for me because there is no way to integrate the simulator in the testing process.

Currently my testing strategy involves writing test programs with which i try to check as many statements as possible. Following is an example of checking arithmetic.

c = a + a;          
if (c != 6) return 4;
c = a + a + a;      
if (c != 9) return 5;
// etc

I am already noticing that it is difficult to find as many corner cases as possible.

Is this a smart way to verify the codegenerator? If so, does anybody know of a pre existing project which includes these kind of tests?

Answer 1

You can do what you want by making your own lit.cfg script with additional definitions to do what you want. For example, I cross build stuff for different targets for my project, ELLCC. I use QEMU to run tests. Part of my modified lit.site.cfg looks like:

config.substitutions.append( ('%microblazeecc', ' ' + config.ecc + ' ' +
                            '-target microblaze-ellcc-linux ') )
config.substitutions.append( ('%microblazeexx', ' ' + config.ecc + '++ ' +
                            '-target microblaze-ellcc-linux ') )
config.substitutions.append( ('%microblazerun', ' ' + ellcc + '/bin/qemu-microblaze ') )

A typical test case looks like:

// Compile and run for every target.
// RUN: %armexx -o %t %s && %armrun %t  | FileCheck -check-prefix=CHECK %s
// RUN: %armebexx -o %t %s && %armebrun %t | FileCheck -check-prefix=CHECK %s
// RUN: %i386exx -o %t %s && %i386run %t | FileCheck -check-prefix=CHECK %s
// RUN: %microblazeexx -o %t %s && %microblazerun %t | FileCheck -check-prefix=CHECK %s
// RUN: %mipsexx -o %t %s && %mipsrun %t | FileCheck -check-prefix=CHECK %s
// RUN: %mipselexx -o %t %s && %mipselrun %t | FileCheck -check-prefix=CHECK %s
// RUN: %ppcexx -o %t %s && %ppcrun %t | FileCheck -check-prefix=CHECK %s
// FAIL: %ppc64exx -o %t %s && %ppc64run %t | FileCheck -check-prefix=CHECK %s
// RUN: %x86_64exx -o %t %s && %x86_64run %t | FileCheck -check-prefix=CHECK %s
// CHECK: foo.i = 10
// CHECK: bye
#include <cstdio>

class Foo {
    int i;
public:
    Foo(int i) : i(i) { }
    int get() { return i; }
    ~Foo() { printf("bye\n"); }
};

int main(int argc, char** argv)
{
    Foo foo(10);
    printf("foo.i = %d\n", foo.get());
}

You can use FileCheck to look for the output you're interested in.

Answer 2

You may want to take a look at the LLVM test suite - a set of "whole programs" that are compiled first with a native platform compiler (e.g. gcc) and then with LLVM and the results are compared.

So if your CPU already has some compiler able to generate code for it, it's a good option. If not, you can rig it so the expected output is generated on your development machine and compared to the output generated by the code compiled with your backend.