temci icon indicating copy to clipboard operation
temci copied to clipboard

Clarify "Number of runs"

Open sgraf812 opened this issue 3 years ago • 0 comments

Here I see the following:

Number of runs

The number of runs per block is either fixed by the runs settings that apply or is between the applying min_runs and max_runs setting. In the latter case, the benchmarking of a program block is stopped early as soon as there is some significance in the benchmarking results compared to all other benchmarked programs.

What does "as soon as there is some significance in the benchmarking results compared to all other benchmarked programs" mean?

Also when I modify the runs setting to some fixed number, I still see

...
Benchmark $min_runs to $max_runs times
...

Even if I never set min_runs or max_runs! I think the actual benchmarking then adheres to the runs setting I did, even if runs is lower than min_runs. For runs=10 and the default setting min_runs=20, max_runs=100, I see

Benchmark 20 to 100 times
Benchmark 1 out of 20 to 100
Benchmark 1 out of 20 to 100
Benchmark 2 out of 20 to 100
Benchmark 3 out of 20 to 100
Benchmark 4 out of 20 to 100
Benchmark 5 out of 20 to 100
Benchmark 6 out of 20 to 100
Benchmark 7 out of 20 to 100
Benchmark 8 out of 20 to 100
Benchmark 9 out of 20 to 100
Benchmark 10 out of 20 to 100
Benchmark 11 out of 20 to 100
Benchmark 12 out of 20 to 100
Report for single runs
...

The excess 1 out of 20 and {11,12} out of 20 lines appear to be related to https://github.com/parttimenerd/temci/issues/136.

sgraf812 avatar Nov 30 '21 09:11 sgraf812