asv

asv copied to clipboard

For some benchmarks, a shift in performance of 1.1 might be significant, while for others a shift of 2 might be insignificant.

A way to specify different factors for different benchmarks would therefore be valuable.

A benchmark attribute seems sensible, but I don't know how difficult it would be to get asv compare to be attribute-aware. I'm also uncertain of the correct order of precedence of global versus benchmark-specific factors in cases where both are present.

For what it's worth: this continues to be a problem for us. I understand that development resource is limited, just wanted to make sure people know that this isn't a 'dead issue'.