hunter icon indicating copy to clipboard operation
hunter copied to clipboard

Extend --magnitude to support specific metrics

Open smccarthy788 opened this issue 3 years ago • 1 comments

Users report wanting to filter different metrics at different magnitudes (e.g. p99 vs p50). However, the --magnitude argument filters across all metrics.

One option could be to allow setting multiple metric=min_magnitude pairs with the --magnitude flag, or exclusively a single float. This would be pretty verbose... Ex:

hunter ... --magnitude p50=0.05 --magnitude p99=0.2

Another option would be setting the min_magnitude in the config for each test's metric, with support for inheritance so each test doesn't need to define it. This option should also support overriding the config.

templates:
  common_metrics:
    metrics:
      p50:
        scale: 1.0e-6
        direction: -1
        min_magnitude: 0.05
      p99:
        scale: 1.0e-6
        direction: -1
        min_magnitude: 0.2
tests:
  local2:
    type: csv
    file: tests/resources/sample.csv
    time_column: time
    metrics:
      m1:
        column: metric1
        direction: 1
      m2:
        column: metric2
        direction: -1
        min_magnitude: 0.99 # Be able to set min_magnitude for specific tests
    attributes: [ commit ]

  remote1:
    inherit: [common_metrics] # min_magnitude set via inheritance
    prefix: "performance_regressions.my_product.%{BRANCH}.test1"

smccarthy788 avatar Dec 02 '21 14:12 smccarthy788