How to write automated unit tests for java annotation processor?

Question

I'm experimenting with java annotation processors. I'm able to write integration tests using the "JavaCompiler" (in fact I'm using "hickory" at the moment). I can run the compile process and analyse the output. The Problem: a single test runs for about half a second even without any code in my annotation processor. This is way too long to using it in TDD style.

Mocking away the dependencies seems very hard for me (I would have to mock out the entire "javax.lang.model.element" package). Have someone succeed to write unit tests for an annotation processor (Java 6)? If not ... what would be your approach?

Answer 1

You're right mocking the annotation processing API (with a mock library like easymock) is painful. I tried this approach and it broke down pretty rapidly. You have to setup to many method call expectations. The tests become unmaintainable.

A state-based test approach worked for me reasonably well. I had to implement the parts of the javax.lang.model.* API I needed for my tests. (That were only < 350 lines of code.)

This is the part of a test to initiate the javax.lang.model objects. After the setup the model should be in the same state as the Java compiler implementation.

DeclaredType typeArgument = declaredType(classElement("returnTypeName"));
DeclaredType validReturnType = declaredType(interfaceElement(GENERATOR_TYPE_NAME), typeArgument);
TypeParameterElement typeParameter = typeParameterElement();
ExecutableElement methodExecutableElement = Model.methodExecutableElement(name, validReturnType, typeParameter);

The static factory methods are defined in the class Model implementing the javax.lang.model.* classes. For example declaredType. (All unsupported operations will throw exceptions.)

public static DeclaredType declaredType(final Element element, final TypeMirror... argumentTypes) {
    return new DeclaredType(){
        @Override public Element asElement() {
            return element;
        }
        @Override public List<? extends TypeMirror> getTypeArguments() {
            return Arrays.asList(argumentTypes);
        }
        @Override public String toString() {
            return format("DeclareTypeModel[element=%s, argumentTypes=%s]",
                    element, Arrays.toString(argumentTypes));
        }
        @Override public <R, P> R accept(TypeVisitor<R, P> v, P p) {
            return v.visitDeclared(this, p);
        }
        @Override public boolean equals(Object obj) { throw new UnsupportedOperationException(); }
        @Override public int hashCode() { throw new UnsupportedOperationException(); }

        @Override public TypeKind getKind() { throw new UnsupportedOperationException(); }
        @Override public TypeMirror getEnclosingType() { throw new UnsupportedOperationException(); }
    };
}

The rest of the test verifies the behavior of the class under test.

Method actual = new Method(environment(), methodExecutableElement);
Method expected = new Method(..);
assertEquals(expected, actual);

You can have a look at the source code of the Quickcheck @Samples and @Iterables source code generator tests. (The code is not optimal, yet. The Method class has to many parameters and the Parameter class is not tested in its own test but as part of the Method test. It should illustrate the approach nevertheless.)

Viel Glück!

Answer 2

This is an old question, but it seems that the state of annotation processor testing hadn't gotten any better, so we released Compile Testing today. The best docs are in package-info.java, but the general idea is that there is a fluent API for testing compilation output when run with an annotation processor. For example,

ASSERT.about(javaSource())
    .that(JavaFileObjects.forResource("HelloWorld.java"))
    .processedWith(new MyAnnotationProcessor())
    .compilesWithoutError()
    .and().generatesSources(JavaFileObjects.forResource("GeneratedHelloWorld.java"));

tests that the processor generates a file that matches GeneratedHelloWorld.java (golden file on the class path). You can also test that the processor produces error output:

JavaFileObject fileObject = JavaFileObjects.forResource("HelloWorld.java");
ASSERT.about(javaSource())
    .that(fileObject)
    .processedWith(new NoHelloWorld())
    .failsToCompile()
    .withErrorContaining("No types named HelloWorld!").in(fileObject).onLine(23).atColumn(5);

This is obviously a lot simpler than mocking and unlike typical integration tests, all of the output is stored in memory.

Answer 3

jOOR is a small Java reflection library that also provides simplified access to the in-memory Java compilation API in javax.tool.JavaCompiler. We added support for this to unit test jOOQ's annotation processors. You can easily write unit tests like this:

@Test
public void testCompileWithAnnotationProcessors() {
    AProcessor p = new AProcessor();

    try {
        Reflect.compile(
            "org.joor.test.FailAnnotationProcessing",
            "package org.joor.test; " +
            "@A " +
            "public class FailAnnotationProcessing { " +
            "}",
            new CompileOptions().processors(p)
        ).create().get();
        Assert.fail();
    }
    catch (ReflectException expected) {
        assertFalse(p.processed);
    }
}

The above example has been taken from this blog post

Answer 4

An option is to bundle all tests in one class. Half a second for compiling etc. is than a constant for a given set of tests, the real test time for a test is egligible, I assume.

Answer 5

I have used http://hg.netbeans.org/core-main/raw-file/default/openide.util.lookup/test/unit/src/org/openide/util/test/AnnotationProcessorTestUtils.java though this is based on java.io.File for simplicity and so has the performance overhead you complain about.

Thomas's suggestion of mocking the whole JSR 269 environment would lead to a pure unit test. You might instead want to write more of an integration test which checks how your processor actually runs inside javac, giving more assurance it is correct, but merely want to avoid disk files. Doing this would require you to write a mock JavaFileManager, which is unfortunately not as easy as it seems and I have no examples handy, but you should not need to mock other things like Element interfaces.

Answer 6

I was in a similar situation, so I created the Avatar library. It won't give you the performance of a pure unit test with no compilation, but if used correctly you shouldn't see much of a performance hit.

Avatar lets you write a source file, annotate it, and convert it to elements in a unit test. This allows you to unit test methods and classes which consume Element objects, without manually invoking javac.