k6 icon indicating copy to clipboard operation
k6 copied to clipboard

Allow more flexible sub-metrics and better UX in thresholds

Open na-- opened this issue 5 years ago • 2 comments

Currently, you can define thresholds on subsets of the metric data points that have specific tag values, like this:

export let options = {
    thresholds: {
        // We want the 95th percentile of request durations for HTTP request duration 
       // metrics tagged with status=200 and staticAsset=yes to be less than 500ms
        "http_req_duration{status:200,staticAsset:yes}": ["p(95)<500"],
    },
};

However, there's no way to be able to specify more complex matching queries that use something besides "equals" and "and". For example, it seems useful to be able to specify something like "the number of HTTP requests with a non-200 status" or "URLs starting with https://www.somesite.com/whatever", etc..

Needless to say, but this should probably be implemented after we refactor the thresholds to have sane parsing (https://github.com/loadimpact/k6/issues/961). https://github.com/loadimpact/k6/issues/1298 is something we should also consider when we decide exactly how to implement this - for example, we should be able to specify thresholds only for 403 responses that were not intended.

Additionally, we should think about refactoring how we configure thresholds. Stuffing complex matching logic in the JSON key seems to be the wrong way to go about it, even now... It may also be worth introducing some JS config-helper functions for whatever we decide, if we can provide a cleaner API that way.

na-- avatar Jan 20 '20 16:01 na--

Something else that might be worth expressing is to be able to automatically generate Rate sub-metrics on the fly from any Counter or Trend metric, based on a such a matching query. Something like this makes sense to me as a threshold: "the rate of HTTP requests to https://somesite.com/api/v1/blah/ with non-200 responses should be less than 1 percent (out of all requests to the same URL)".

Unfortunately I don't currently have even the vaguest suggestion how to configure something like this in a same manner... So, given that it (and the things above) could be expressed with some extra coding via custom metrics, it's not urgent or a must-have feature. But it would be very nice if our threshold rule engine was powerful enough that you are able to encode such rules in a declarative manner in a single place of your script...

na-- avatar Jan 20 '20 17:01 na--

Another use case for this is submetrics for nested groups: https://github.com/grafana/k6/issues/2309

na-- avatar Dec 22 '21 11:12 na--