YAuB - Yet another Micro Benchmark

Note -> Follow-on question posted here: YAuB - Micro Benchmark Follow-on

I recently answered this Beautiful String question, where the task is to maximize the value of a string by assigning each character a different value (from 1 to 26) in a way that the sum of all characters is maxmized.

A number of different solutions were presented, but, the question is, which one is fastest?

This inspired me to write yet another micro-benchmark system that allows you to run and get metrics on simple code runs, using some Java 8 tricks to make the invocation easy.

For example, in the question/answer set above, there are the following suggestions:

• Legato has 2 implementations in his question

• Janos has one

• I have a few

That makes 6 implementations to sort through. It would be really nice to be able to do:

• create an instance of a benchmark system.

• add some tasks to run

• run the tasks with specified constraints

• get a report of the significant statistics for each task.

With that in mind, the goal is to allow the following:

```
final String line = "This is a test, including punctuation, and other words"
+ " and numbers like 1, UPPER, and Lower letters";
final int expect = 1574;

UBench uBench = new UBench("Beautiful");

uBench.addTask(Task.buildCheckedIntTask("Legato Java7", () -> getMaximumBeauty(line), expect));
uBench.addTask(Task.buildCheckedIntTask("Legato Java8", () -> getMaximumBeauty8(line), expect));
uBench.addTask(Task.buildCheckedIntTask("Janos Java7", () -> computeMaxBeauty(line), expect));
uBench.addTask(Task.buildCheckedIntTask("Rolfl Java7", () -> beautyMax7(line), expect));
uBench.addTask(Task.buildCheckedIntTask("Rolfl Java8Regex", () -> beautyMaxF(line), expect));
uBench.addTask(Task.buildCheckedIntTask("Rolfl Java8Filter", () -> beautyMax8(line), expect));

System.out.println("Warming up");
uBench.benchMark(5000).stream().forEach(System.out::println);
System.out.println("\n\nReal runs\n\n");
uBench.benchMark(10000).stream().sorted(

From a usability perspective, this is a bit weird...

public static final Task buildCheckedIntTask(final String name, final IntSupplier benchmark, final int expect) {
    return new Task(name) {

There's not really any need for a Task here, it could just as well be Task.

Additionally, as you know what type it returns, I see no reason to return Task. It would be beneficial to return Task here IMO.

I would expect the checking to be done within the check method, and not within perform(), like the following:

public static final Task buildCheckedIntTask(final String name, final IntSupplier benchmark, final int expect) {
    return new Task(name) {
            @Override
            protected Integer perform() throws Exception {
                int got = benchmark.getAsInt();
                return got;
            }

            @Override
            protected boolean check(Integer result) {
                return result == expect;
            }

About this buildVoidTask, why use the Object type when you can use the Void type?

public static final Task buildVoidTask(final String name, final Runnable function) {
    return new Task(name) {
        @Override
        protected Void perform() throws Exception {
            function.run();
            return null;
        }

        @Override
        protected boolean check(Object result) {
            return true;
        }

When it comes to abstract classes, I sometimes move the abstract methods themselves to an interface, make the abstract class a concrete class, and pass it an object of the interface. Like the following:

public interface TaskCaller {
    R getResult() throws Exception;
    boolean checkResult(R result);
}

Although this can easily be separated into ThrowingSupplier (you'd have to make that one) and Predicate

public class Task {
    private final ThrowingSupplier supplier;
    private final Predicate checker;

    public Task(String name, ThrowingSupplier supplier, Predicate predicate) {
        this.name = name;
        this.supplier = supplier;
        this.checker = predicate;
    }
    ...
}

It is good that you at the moment allow extending Task, but you haven't provided an easy way to use a Predicate for the checking. Let's say you have a PrimeNumberGenerator for example. It's not possible to provide a single Long expected to that. It is however possible to verify that the generated numbers is prime.

If you were to use this, your separate build methods could be drastically shortened:

public static final  Task buildCheckedTask(final String name, final Supplier benchmark, final T expect) {
        return new Task(name, benchmark, r -> Objects.equals(r, expect));
    }

private final List> tasks = new ArrayList<>();

It might just be me, but I believe instead of using the synchronization throughout your class you could wrap this one in Collections.synchronizedList (although you know more about multi-threaded things than I do, there might be a significant performance drawback by using this?)

/**
 * Benchmark all tasks until it they complete the desired elapsed time
 * @param iterations
 *            number of iterations to run.
 * @return the results of all completed tasks.
 */
public List benchMark(final long timeLimit, final TimeUnit timeUnit) {

• Avoid grammatical errors in JavaDoc. That's not useful ;) until it they...

• The @param does not match the actual parameters for this method

} catch (Exception e) {
    throw new IllegalStateException(String.format("Failed execution in %s with %s", name, e.getMessage()), e);
}

Rethrowing an exception, good, nothing wrong with that... but... Let's say that a simple new RuntimeException() is thrown somewhere inside the code. Then the message for this IllegalStateException will be:


Failed execution in MyTask with null

That's right. With what? Instead of using e.getMessage(), it is a lot more helpful to simply use e.

Consider this code:

System.out.println(new RuntimeException("test error"));
System.out.println(new RuntimeException("test error").getMessage());

System.out.println(new RuntimeException());
System.out.println(new RuntimeException().getMessage());

The output here is:

java.lang.RuntimeException: test error
test error
java.lang.RuntimeException
null

Using .getMessage() drastically reduces the usefulness when used in a string.

Other micro-benchmark frameworks have a feature for scanning a class for @Benchmark annotations. [meta-tag:feature-request]

Also, probably the most important thing with regards to usability, make it easy to use your library as a dependency! At the moment, there does not seem to be a build.gradle, pom.xml or similar for your project. If there was a Maven dependency available for your project, doesn't necessarily have to be on Maven Central, anyone could use it in an easy way.

StackExchange Code Review Q#82325, answer score: 7