criterion
criterion copied to clipboard
haskell
Make it easier to compare different benchmarks
When comparing multiple different benchmarks, it would be nice to have a way to mark them as "comparable" (perhaps bcomparisongroup), with the following effects:
- numbers are all generated on the same time scale (e.g. all 'μs' or all 'ms', not a mixture)
- graphs are produced with the same vertical scale
This would make it easier to eyeball differences.
As an extension, perhaps also the library could do some statistics to see if there are true differences between the timings (ANOVA? it's been a while...), but that would be an added bonus.
Just wonder in relation to this issue, whether there is any way to "scale" the PDF values given by criterion report?
Context: I am running a lot of individual benchmarks and would like to compare them on a KDE graph I am trying to render using the KDE information in an exported JSON report. However, since the y-value scales are inconsistent it is hard to compare different benchmarks. If there was a way to even make these scales even roughly comparable, that would be extremely helpful.
EDIT It would be useful to for example pass in a range [0,1] such that alll the kdePDf values are rescaled to fall within this value range
numbers are all generated on the same time scale (e.g. all 'μs' or all 'ms', not a mixture)
Yes please
I was recently made aware of the criterion-compare tool. I'd be willing to incorporate similar functionality in criterion, should someone want to prepare a patch for it.
