ion-python icon indicating copy to clipboard operation
ion-python copied to clipboard

Benchmark CLI `compare` can have inaccurate results for some metrics

Open popematt opened this issue 2 years ago • 0 comments

The compare command in the benchmark CLI always assumes lower is better. This is not true when measuring e.g. ops per second, and so the compare command can erroneously detect regressions for those metrics.

popematt avatar Aug 01 '23 00:08 popematt