benchmark icon indicating copy to clipboard operation
benchmark copied to clipboard

[FR] Determine suitable time unit for a benchmark if none is set explicitly

Open Matthdonau opened this issue 2 years ago • 1 comments

Is your feature request related to a problem? Please describe. Setting a suitable time unit for a specific benchmark is sometimes tedious when the machine the benchmark is developed on is not the only machine the benchmark is run on. E.g imagine a company where code is written for high performant server cpus as well as for weak embedded cpus. Furthermore if some code is sped up for one reason or another the benchmark time unit for this code might not get adapted.... This may end up in results which are hard to interpret on a certain machine because the set time unit is too low and the reported time is as very large number. Furthermore I would argue, that if the set time unit is way to small (often by default --> ns) the results pretend to be very precise while they aren't. Imagine a benchmark which takes a billion ns. This benchmark will print results down to a ns.

Describe the solution you'd like I would like libbenchmark to figure out a suitable time uint itself (not for benchmarks which have set their unit specifically or through the global flag). This could be done by e.g running the benchmark and then choose a time unit in which the result is precise down to 0.1% of the total runtime. E.g running a benchmark which takes 10051ms should be reported as 10.05s. However for benchmarks which run for e.g different ranges the time unit should be choosen the same for all.

Describe alternatives you've considered I am not aware of any.

Matthdonau avatar Jun 02 '22 17:06 Matthdonau

i think we used to do something like this...

i vaguely remember some confusion as some benchmarks would be in 'ns', some in 'ms' and when read next to each other it's easy to miss the units, which is why we went with the explicit time unit.

i'd be ok with an 'auto' option on the flag though, if you think that would be useful.

dmah42 avatar Jun 06 '22 09:06 dmah42