Unit testing annotation processors in Java

I'm working on a micro-benchmarking framework in Java.
To give you some context, this is how it will work:

-
Given multiple implementations of some algorithm, and you want to compare which one performs better by measuring the processing time.

-
Create a brand new empty class, annotate with @Benchmark (from the framework)

-
Add a public void method, that simply calls one of the implementations you want to measure, with all the necessary parameters.
Annotate the method with @MeasureTime (from the framework).
Repeat for each implementation.

-
Make sure that all methods work with the same input sets,
and that they don't produce side effects.

-
Run the framework

The annotations @Benchmark and @MeasureTime can take parameters:

• @Benchmark.iterations: the number of times to repeat all @MeasureTime methods by default

• @Benchmark.warmUpIterations: the number of warmup runs, which won't be included in the time measurements, for all @MeasureTime methods by default

• @MeasureTime.iterations: the number of times to repeat the method, overriding the default

• @MeasureTime.warmUpIterations: as the name suggests, overriding the default

I want to test that the framework is using the annotation parameters correctly.
That is:

• the default iterations and warm-up iterations are correct (1 and 0)

• the parameters of @Benchmark correctly override the defaults

• the parameters of @MeasureTime correctly override the defaults

Here's a test for the most simple case:

public class DefaultRunnerTest {

    public static int runCount = 0;

    @Benchmark
    public static class RunWithDefaults {
        @MeasureTime
        public void sample() {
            ++runCount;
        }
    }

    @Before
    public void setUp() {
        runCount = 0;
    }

    @Test
    public void testRunWithDefaults() {
        new DefaultRunner(RunWithDefaults.class).run();
        assertEquals(1, runCount);
    }
}

What I don't like is the runCount stat

There's one particular thing that I don't like about your framework, which will help you get rid of that ugly static variable:

You're passing a Class to the framework instead of an instance!

Don't put the responsibility of creating an actual object to test the methods on on the Framework, let the user supply an instance of the object to your framework. This will allow the user to use Dependency-Injection style constructors for the objects that are going to be tested, in this case this would allow you to do:

private final AtomicInteger runCount = new AtomicInteger();

@Test
public void testRunWithDefaults() {
    new DefaultRunner(new RunWithDefaults(runCount)).run();
    assertEquals(1, runCount.get());
}

In your framework to get the class (and then to find the methods and such), you just call obj.getClass()

A few other comments:

Even though it's not really 'up for review', there are a few comments I have about your framework class:

public DefaultRunner(Class klass) {

Here you should use generics: public DefaultRunner(Class klass) {

Although with the suggested changes above, this would be:

public DefaultRunner(Object object) {
    this.klass = object.getClass();

It is more common to use the name clazz for a class. 'klass' sounds like the word 'class' translated to another language. (It is actually how one would translate it to Swedish)

Annotation annotation = klass.getAnnotation(Benchmark.class);

Would be better of as:

Benchmark annotation = klass.getAnnotation(Benchmark.class);

So that you don't have to typecast it later on. getAnnotation uses a generic return type so you can make use of it directly.

This would make the code for your constructor as this:

private final Class clazz;

public DefaultRunner(Object obj) {
    this.clazz = obj.getClass();

    Benchmark annotation = klass.getAnnotation(Benchmark.class);

Of course, if you still want to, you could provide an overloaded constructor that requires a Class and tries to instantiate that, although I think it is better to stick with one way and this seems to be the best way IMO, with regards to flexibility, dependency injection, and a whole lot of other things.

StackExchange Code Review Q#67099, answer score: 7