BurntSushi
feat: allow to define one argument as the baseline for comparison
Feature request: Define the first 'baseline' passed (-b option) to be the one used for calculating speedups.
Example, this command
critcmp -b before change1 change2 change3
compares speedup of the baselines called changeN against the before baseline.
group before change1 change2 change3
----- ------- ------- ------- -------
sort/10 1.00 38.2±0.56ns 1.39 53.1±0.60ns 1.59 13.1±0.60ns 0.75 8.1±0.60ns
Currently, the output highlights with a color the fastest implementation in each group.
+1. I'd much rather consistently scan down a single column and see whether the numbers are greater or less than 1 (and maybe even use something like awk to compute the geometric means of the non-before columns) than scan left and right to see which column is 1.00 and then compare the others to it.
Strong +1 for built-in regression detection. I'm really confused why this feature was absent. For CI integration it'd be really helpful to easily detect if the regression has happened. Right now the tool is hard to use for regression detection because the only indication of which run is faster is just the color which is hard/annoying to grep.
Strong +1 for built-in regression detection. I'm really confused why this feature was absent.
It isn't. If you pass -t10 to critcmp for example, then it will hide any comparisons under 10% difference. So it is totally false that color is the only indicator here.
Also, I don't use critcmp in CI. I use it primarily interactively to analyze benchmark outputs. If you want something for CI, you might consider a different tool (and perhaps not using wall clock time at all).
In case it isn't obvious, future "+1" comments are not welcome. They are annoying and don't really add anything to the conversation. I personally don't have any plans to work on this at present, but someone else is welcome to submit a patch. If it's small and simple, I'll work to get it merged.
@BurntSushi The threshold option only prints the differences above the threshold, but it prints both regressions and performance improvements, and I'm not sure there is a way to distinguish between these two using the existing cli options of critcmp..?
Yeah it'd be annoying, but you'd have to parse out whether there's an improvement or regression. Or you could flag any big change as something worthy of human attention. (I've had cases where a big benchmark improvement indicated a bug.)