use more rigourous statistics

[dgryski]

Russ Cox has written https://godoc.org/rsc.io/benchstat which does the same thing but with better math.

[BurntSushi]

Yes, I've seen that and I definitely want to add better math.

[mgeisler]

This might not be necessary. As far as I understand, go test -bench does not do any kind of statistics itself -- it just runs the benchmark code repeated until the running time exceeds 1 sec (by default) and outputs the average: https://golang.org/src/testing/benchmark.go

So this means that the Go world needs extra tools to do statistics on multiple runs. However, that is not the case for cargo bench which comes with some builtin statistics and outlier detection: https://github.com/rust-lang/rust/blob/master/src/libtest/stats.rs#L308

This suggests to me that cargo benchcmp should be careful with comparing values that are already filtered and adjusted by cargo bench -- similar to how an average of an average doesn't give you a proper average of the underlying values. Perhaps cargo bench itself could be improved with more fancy statistic -- I don't know much about that since I'm not a statistician :-)