promtool: Allow inexact matches in exp_alerts

[carlpett]

Proposal

Use case. Why is this important? Currently, when building a rule test, one needs the exp_alerts to be exact matches of the alert that would fire, with all labels and annotations. Since these can be fairly rich in real-world usage, I propose that matching can be "inexact". Rather than all labels and annotations being exact matches, only those actually present in the exp_labels and exp_annotations must match. I frequently come across organisations where entire runbooks are written in annotations, and duplicating those into each rule adds a lot of friction to test adoption. Fixing a typo in the runbook breaking unit tests, for example.

This could be made opt-in with a matching_mode or such, if there is a frequent desire to match all the things.

Alternatively, or perhap as a complement, being able to check for presence but ignoring content. This could facilitate something like a test that validates that all alerting rules have a runbook, for example.

[beorn7]

Hello from the bug scrub.

This sounds useful to us, but nobody worked on it in 4y. @dgl can you confirm that this is still relevant and that we would welcome help implementing it?

[aurifi]

Hey @beorn7 ! I also think this feature would vastly improve testing rules. If there currently are no different plans I might give it a shot. :)

[beorn7]

I don't know of any other plans or people working on this. I'll assign this to you @szczepad .

One more ping for @dgl in case there is something to say.

[dgl]

Hi, happy for you to work on this @szczepad -- the main issue with the approach in #11033 was it used a YAML specific feature, i.e. it wouldn't be able to roundtrip as JSON. The design issue is that approach with a dynamic field solved the problem of what happens if there's both a regex and an exact match while keeping backwards compatibility.

Probably the best approach is we have something like:

exp_annotations_regex:
  some-field: 'a-regex-.*'
exp_annotations:
  other-field: "fixed-value"

promtool would then require all labels be present in at least one of those (both should be ok, as that allows some basic YAML inheritance to e.g. define a regex for a bunch of alerts and override the actual value in some alert tests).

[aurifi]

Hey @dgl , thanks for the message! :) Just making sure that we have the same understanding. From what I understood the purpose of this issue is to be able to not have to consider all labels and annotations an alert might have.

So if I have an Alert firing like

alert: HighRequestLatency
  labels:
    severity: critical
    team: backend
  annotations:
    summary: High request latency

I might want to assert that this alert has the label 'team' set to 'backend' but explicitely don't want to assert anything on summary, as this might freely change. The current way would be to set all labels and annotations, which makes the unit-tests quite verbose and requires to change the unit-tests on every change, although only informational data might be added.

Being able to match on regex would also totally make sense. But right now I would see this as a seperate topic to this one. However I didn't have the chance to look into the code for this issue yet, so your comment might be clearer to me as soon as this has happened. :D