hunter
hunter copied to clipboard
Extend --magnitude to support specific metrics
Users report wanting to filter different metrics at different magnitudes (e.g. p99 vs p50). However, the --magnitude
argument filters across all metrics.
One option could be to allow setting multiple metric=min_magnitude
pairs with the --magnitude
flag, or exclusively a single float. This would be pretty verbose... Ex:
hunter ... --magnitude p50=0.05 --magnitude p99=0.2
Another option would be setting the min_magnitude
in the config for each test's metric, with support for inheritance so each test doesn't need to define it. This option should also support overriding the config.
templates:
common_metrics:
metrics:
p50:
scale: 1.0e-6
direction: -1
min_magnitude: 0.05
p99:
scale: 1.0e-6
direction: -1
min_magnitude: 0.2
tests:
local2:
type: csv
file: tests/resources/sample.csv
time_column: time
metrics:
m1:
column: metric1
direction: 1
m2:
column: metric2
direction: -1
min_magnitude: 0.99 # Be able to set min_magnitude for specific tests
attributes: [ commit ]
remote1:
inherit: [common_metrics] # min_magnitude set via inheritance
prefix: "performance_regressions.my_product.%{BRANCH}.test1"