Micro-Benchmark framework

Performance benchmarking small functions in Java is notoriously difficult, and there are a number of tools out there to help (caliper, others). Those other tools require a fair amount of setup and installation to get them working.

This code is adapted from a question @skiwi asked. He produced this ProjectEuler framework, and I have adapted it to be more performance-monitoring friendly.

The basic premise is that you want to run a method multiple times. Some of those are warmup times, and the others are 'real' runs. The tool performs the warmup, then later does the real run. It averages the execution times for the real run to produce the performance time for the method.

The bulk of the logic is incorporated in to a class called Problem which has an execute() method that is abstract.

package euler;

public abstract class Problem {
    private final String name;
    private final int warmup;
    private final int realruns;

    public Problem(String name, int warmups, int realruns) {
        this.name = name;
        this.warmup = warmups;
        this.realruns = realruns;
    }

    public String getResult() {
        return String.valueOf(execute());
    }

    public final int getWarmups() {
        return warmup;
    }

    public final int getRealRuns() {
        return realruns;
    }

    public final String getName() {
        return name;
    }

    public abstract T execute();

}

A typical implementation of this Problem, for example, is to calculate the average of an array of integers, and it would be implemented as:

public class AverageIntegers extends Problem{

    private static final int[] DATA = {1, 2, 3, 4, 5, 6, 7, 8, 9, 8, 7, 6, 5, 4, 3, 2, 1};

    public AverageIntegers() {
        super("Average Integers", 1000, 10000);
    }

    @Override
    public Double execute() {
        int sum = 0;
        for (int v : DATA) {
            sum += v;
        }
        return sum / (double)DATA.length;
    }
}

With the above implemen

Framework

/* **********************************
     * ADD YOUR PROBLEMS HERE!
     * ***********************************/

I'm sorry, did you just call this a framework? Never in my life have I seen a framework where I had to edit the sourcecode to make it usable.

Let users of the framework pass the List> problems to the constructor instead.

And while you're at it, can I please have a public method from the ProjectEuler class? process for example, it'd be ideal as a public method.

Naming

The ProjectEuler class isn't restricted to only problems that implements the EulerProblem interface, is it? I'm sure you can find a better name. ProblemBenchmarker perhaps?

Abstract class?

I don't see the point of making Problem an abstract class. Personally, I think it makes more sense to add a field to the Problem class and a parameter to the constructor:

public Problem(String name, int warmups, int realruns, Supplier supplier) {

And simply modify the execute method to:

public T execute() {
    return supplier.get();
}

This will provide the possibility to have multiple problems in the same class, which IMO provides a nice overview of the problems:

public static void main(String[] args) {
    List> problems = new ArrayList<>();
    problems.add(new Problem<>("Averaging", 1000, 10000, ProblemMain::problemOne));
    problems.add(new Problem<>("Multiplying", 1000, 10000, ProblemMain::problemTwo));
    
    ProjectEuler euler = new ProjectEuler(problems);
       
    euler.process();
    System.out.println("\n\nWarmup Complete\n\n");
    euler.process();

}

private static final int[] DATA = {1, 2, 3, 4, 5, 6, 7, 8, 9, 8, 7, 6, 5, 4, 3, 2, 1};

public static Double problemOne() {
    int sum = 0;
    for (int v : DATA) {
        sum += v;
    }
    return sum / (double)DATA.length;
}

public static Integer problemTwo() {
    int sum = 0;
    for (int v : DATA) {
        sum *= v;
    }
    return sum;
}

Generics

I'm not so sure that the generics of the Problem class does you any good. It's used in a List> anyway so I don't see that the type safety of generics gives you anything. Consider removing the generics and use a Supplier inside it instead, and have the execute() method return an Object.

Of course it'd be neat if it was possible to avoid the auto-boxing that Java does on primitive values, but I expect that would require some code duplication, and I'm not sure if the potential performance gain you could get out of it is worth the code to add it. (I bet you can answer that better than I can, but you have taught me that auto-boxing does affect the performance a bit)

Usefulness

Overall, I think this code will be extremely useful, especially when I can use it without having to modify it's source!

StackExchange Code Review Q#52289, answer score: 8