HiveBrain v1.2.0
Get Started
← Back to all entries
patternMinor

Why do benchmark results vary at all?

Submitted by: @import:stackexchange-cs··
0
Viewed 0 times
whyallvarybenchmarkresults

Problem

Benchmarking typically consists of getting the current CPU time, executing test code a large number of times, and then subtracting the new CPU time from the previous one. However when you benchmark code multiple times, the results tend to vary either slightly to significantly each time. You will very likely not get the exact same number each time even for the same code and same number of iterations.

However, CPUs have a fixed clock rate (do they not?), and instructions typically take the same amount of time to execute for the same conditions (add, mov, or, etc. typically take a fixed integer number of clock cycles or do they not?), and the same instructions are being executed the same number of times under seemingly the same conditions.

So in theory every benchmark should return the exact same number, yet clearly they do not. What goes on behind the scenes in a CPU that could cause variations in benchmark results for the exact same benchmark?

Solution

Your assumption that the clock rate is constant is wrong. My computer at home can at any time switch each core to one of 15 different clock speeds.

Your assumption that the same operation always takes the same time is wrong. The total processor state is extremely complicated and not exactly reproducible. The benchmark runs on a computer that does other things that can and will affect the benchmark times.

Context

StackExchange Computer Science Q#167243, answer score: 5

Revisions (0)

No revisions yet.