critcmp
critcmp copied to clipboard
BurntSushi
human sort the benchmarks
We have a set of micro benchmarks which we compare using critcmp. This would be an example benchmark result:
group benchmark-current.baseline benchmark-main.baseline
----- -------------------------- -----------------------
fibonacci_loop/our/1 1.06 196.4±0.37ns 4.9 MElem/sec 1.00 185.8±4.38ns 5.1 MElem/sec
fibonacci_loop/our/1024 1.01 78.3±0.07µs 12.5 MElem/sec 1.00 77.5±0.09µs 12.6 MElem/sec
fibonacci_loop/our/1048576 1.01 80.5±0.27ms 12.4 MElem/sec 1.00 79.9±0.21ms 12.5 MElem/sec
fibonacci_loop/our/128 1.00 9.9±0.05µs 12.4 MElem/sec 1.00 9.8±0.01µs 12.4 MElem/sec
fibonacci_loop/our/131072 1.00 10.0±0.01ms 12.6 MElem/sec 1.01 10.1±0.01ms 12.4 MElem/sec
fibonacci_loop/our/16 1.00 1346.4±1.96ns 11.3 MElem/sec 1.00 1352.6±1.23ns 11.3 MElem/sec
fibonacci_loop/our/16384 1.01 1246.7±1.13µs 12.5 MElem/sec 1.00 1237.5±1.13µs 12.6 MElem/sec
fibonacci_loop/our/2 1.04 282.9±1.63ns 6.7 MElem/sec 1.00 271.3±2.37ns 7.0 MElem/sec
fibonacci_loop/our/2048 1.02 158.5±0.15µs 12.3 MElem/sec 1.00 154.8±0.14µs 12.6 MElem/sec
fibonacci_loop/our/256 1.01 19.6±0.02µs 12.4 MElem/sec 1.00 19.5±0.03µs 12.5 MElem/sec
fibonacci_loop/our/262144 1.00 19.9±0.04ms 12.5 MElem/sec 1.01 20.1±0.02ms 12.5 MElem/sec
fibonacci_loop/our/32 1.01 2.6±0.00µs 11.9 MElem/sec 1.00 2.5±0.00µs 12.0 MElem/sec
fibonacci_loop/our/32768 1.01 2.5±0.00ms 12.5 MElem/sec 1.00 2.5±0.00ms 12.6 MElem/sec
fibonacci_loop/our/4 1.01 433.6±0.98ns 8.8 MElem/sec 1.00 428.4±0.66ns 8.9 MElem/sec
fibonacci_loop/our/4096 1.04 321.8±36.30µs 12.1 MElem/sec 1.00 309.5±0.38µs 12.6 MElem/sec
fibonacci_loop/our/512 1.01 39.2±0.04µs 12.5 MElem/sec 1.00 38.8±0.05µs 12.6 MElem/sec
fibonacci_loop/our/524288 1.00 39.8±0.02ms 12.5 MElem/sec 1.00 39.8±0.04ms 12.6 MElem/sec
fibonacci_loop/our/64 1.01 5.0±0.01µs 12.1 MElem/sec 1.00 5.0±0.01µs 12.3 MElem/sec
fibonacci_loop/our/65536 1.03 5.1±0.00ms 12.3 MElem/sec 1.00 4.9±0.00ms 12.6 MElem/sec
fibonacci_loop/our/8 1.02 742.2±1.06ns 10.3 MElem/sec 1.00 727.0±0.90ns 10.5 MElem/sec
fibonacci_loop/our/8192 1.01 623.9±1.02µs 12.5 MElem/sec 1.00 619.0±0.59µs 12.6 MElem/sec
fibonacci_recursive/our/1 1.00 222.6±1.16ns 4.3 MElem/sec 1.15 255.8±13.62ns 3.7 MElem/sec
fibonacci_recursive/our/128 1.00 11.3±0.02µs 10.8 MElem/sec 1.21 13.6±0.26µs 8.9 MElem/sec
fibonacci_recursive/our/16 1.00 1735.4±3.49ns 8.8 MElem/sec 1.11 1932.2±45.86ns 7.9 MElem/sec
fibonacci_recursive/our/2 1.00 342.7±2.06ns 5.6 MElem/sec 1.09 371.9±13.99ns 5.1 MElem/sec
fibonacci_recursive/our/256 1.00 22.2±0.04µs 11.0 MElem/sec 1.35 29.8±0.97µs 8.2 MElem/sec
fibonacci_recursive/our/32 1.00 3.1±0.01µs 9.7 MElem/sec 1.32 4.1±0.06µs 7.4 MElem/sec
fibonacci_recursive/our/4 1.00 561.4±4.24ns 6.8 MElem/sec 1.09 610.3±24.04ns 6.3 MElem/sec
fibonacci_recursive/our/512 1.00 43.7±0.06µs 11.2 MElem/sec 1.32 57.8±0.85µs 8.4 MElem/sec
fibonacci_recursive/our/64 1.00 5.9±0.01µs 10.4 MElem/sec 1.31 7.7±0.56µs 8.0 MElem/sec
fibonacci_recursive/our/8 1.00 951.8±1.61ns 8.0 MElem/sec 1.12 1070.3±23.31ns 7.1 MElem/sec
Unfortunately, the order of the groups is lexically sorted (I believe?), but it would be way nicer for it to be human-readable numerically sorted:
group benchmark-current.baseline benchmark-main.baseline
----- -------------------------- -----------------------
fibonacci_loop/our/1 1.06 196.4±0.37ns 4.9 MElem/sec 1.00 185.8±4.38ns 5.1 MElem/sec
fibonacci_loop/our/2 1.04 282.9±1.63ns 6.7 MElem/sec 1.00 271.3±2.37ns 7.0 MElem/sec
fibonacci_loop/our/4 1.01 433.6±0.98ns 8.8 MElem/sec 1.00 428.4±0.66ns 8.9 MElem/sec
fibonacci_loop/our/8 1.02 742.2±1.06ns 10.3 MElem/sec 1.00 727.0±0.90ns 10.5 MElem/sec
fibonacci_loop/our/16 1.00 1346.4±1.96ns 11.3 MElem/sec 1.00 1352.6±1.23ns 11.3 MElem/sec
fibonacci_loop/our/32 1.01 2.6±0.00µs 11.9 MElem/sec 1.00 2.5±0.00µs 12.0 MElem/sec
fibonacci_loop/our/64 1.01 5.0±0.01µs 12.1 MElem/sec 1.00 5.0±0.01µs 12.3 MElem/sec
fibonacci_loop/our/128 1.00 9.9±0.05µs 12.4 MElem/sec 1.00 9.8±0.01µs 12.4 MElem/sec
fibonacci_loop/our/256 1.01 19.6±0.02µs 12.4 MElem/sec 1.00 19.5±0.03µs 12.5 MElem/sec
fibonacci_loop/our/512 1.01 39.2±0.04µs 12.5 MElem/sec 1.00 38.8±0.05µs 12.6 MElem/sec
fibonacci_loop/our/1024 1.01 78.3±0.07µs 12.5 MElem/sec 1.00 77.5±0.09µs 12.6 MElem/sec
fibonacci_loop/our/2048 1.02 158.5±0.15µs 12.3 MElem/sec 1.00 154.8±0.14µs 12.6 MElem/sec
fibonacci_loop/our/4096 1.04 321.8±36.30µs 12.1 MElem/sec 1.00 309.5±0.38µs 12.6 MElem/sec
fibonacci_loop/our/8192 1.01 623.9±1.02µs 12.5 MElem/sec 1.00 619.0±0.59µs 12.6 MElem/sec
fibonacci_loop/our/16384 1.01 1246.7±1.13µs 12.5 MElem/sec 1.00 1237.5±1.13µs 12.6 MElem/sec
fibonacci_loop/our/32768 1.01 2.5±0.00ms 12.5 MElem/sec 1.00 2.5±0.00ms 12.6 MElem/sec
fibonacci_loop/our/65536 1.03 5.1±0.00ms 12.3 MElem/sec 1.00 4.9±0.00ms 12.6 MElem/sec
fibonacci_loop/our/131072 1.00 10.0±0.01ms 12.6 MElem/sec 1.01 10.1±0.01ms 12.4 MElem/sec
fibonacci_loop/our/262144 1.00 19.9±0.04ms 12.5 MElem/sec 1.01 20.1±0.02ms 12.5 MElem/sec
fibonacci_loop/our/524288 1.00 39.8±0.02ms 12.5 MElem/sec 1.00 39.8±0.04ms 12.6 MElem/sec
fibonacci_loop/our/1048576 1.01 80.5±0.27ms 12.4 MElem/sec 1.00 79.9±0.21ms 12.5 MElem/sec
fibonacci_recursive/our/1 1.00 222.6±1.16ns 4.3 MElem/sec 1.15 255.8±13.62ns 3.7 MElem/sec
fibonacci_recursive/our/2 1.00 342.7±2.06ns 5.6 MElem/sec 1.09 371.9±13.99ns 5.1 MElem/sec
fibonacci_recursive/our/4 1.00 561.4±4.24ns 6.8 MElem/sec 1.09 610.3±24.04ns 6.3 MElem/sec
fibonacci_recursive/our/8 1.00 951.8±1.61ns 8.0 MElem/sec 1.12 1070.3±23.31ns 7.1 MElem/sec
fibonacci_recursive/our/16 1.00 1735.4±3.49ns 8.8 MElem/sec 1.11 1932.2±45.86ns 7.9 MElem/sec
fibonacci_recursive/our/32 1.00 3.1±0.01µs 9.7 MElem/sec 1.32 4.1±0.06µs 7.4 MElem/sec
fibonacci_recursive/our/64 1.00 5.9±0.01µs 10.4 MElem/sec 1.31 7.7±0.56µs 8.0 MElem/sec
fibonacci_recursive/our/128 1.00 11.3±0.02µs 10.8 MElem/sec 1.21 13.6±0.26µs 8.9 MElem/sec
fibonacci_recursive/our/256 1.00 22.2±0.04µs 11.0 MElem/sec 1.35 29.8±0.97µs 8.2 MElem/sec
fibonacci_recursive/our/512 1.00 43.7±0.06µs 11.2 MElem/sec 1.32 57.8±0.85µs 8.4 MElem/sec
(this output was created using sort --stable --field-separator=/ --key=1,2d --key=3g plus manual fixing of the first two lines).
