temci
temci copied to clipboard
Clarify "Number of runs"
Here I see the following:
Number of runs
The number of runs per block is either fixed by the
runs
settings that apply or is between the applyingmin_runs
andmax_runs
setting. In the latter case, the benchmarking of a program block is stopped early as soon as there is some significance in the benchmarking results compared to all other benchmarked programs.
What does "as soon as there is some significance in the benchmarking results compared to all other benchmarked programs" mean?
Also when I modify the runs
setting to some fixed number, I still see
...
Benchmark $min_runs to $max_runs times
...
Even if I never set min_runs
or max_runs
! I think the actual benchmarking then adheres to the runs
setting I did, even if runs
is lower than min_runs
. For runs=10
and the default setting min_runs=20
, max_runs=100
, I see
Benchmark 20 to 100 times
Benchmark 1 out of 20 to 100
Benchmark 1 out of 20 to 100
Benchmark 2 out of 20 to 100
Benchmark 3 out of 20 to 100
Benchmark 4 out of 20 to 100
Benchmark 5 out of 20 to 100
Benchmark 6 out of 20 to 100
Benchmark 7 out of 20 to 100
Benchmark 8 out of 20 to 100
Benchmark 9 out of 20 to 100
Benchmark 10 out of 20 to 100
Benchmark 11 out of 20 to 100
Benchmark 12 out of 20 to 100
Report for single runs
...
The excess 1 out of 20
and {11,12} out of 20
lines appear to be related to https://github.com/parttimenerd/temci/issues/136.