alertmanager icon indicating copy to clipboard operation
alertmanager copied to clipboard

Documentation for inhibit rules is not clear.

Open Freyert opened this issue 3 years ago • 3 comments

What did you do? I'm trying to use inhibit rules for the first time. After reading the documentation many times I have no idea why there is both a target alert and a source alert. Why is there not just a single target alert that I'm trying to inhibit?

I've reviewed many mailing list conversations, GH issues, and Stack Overflow posts regarding inhibit rules, but no one seems to understand how they work. Answerers present new configuration to try, but never display a deep understanding of the thinking behind inhibit rules themselves.

I think also that alert_relabel_configs are a bit more straightforward to use if you just want to drop alerts with particular labels.

Update: I managed to find a mailing list post that explains inhibit_rules in a way that was easier to grasp for me than the documentation. https://groups.google.com/g/prometheus-users/c/jPy3uqa83no/m/JqlowJBQAgAJ

Inhibit rules specify how one alert can inhibit (~silence) another – say, you have an alert that says "the whole datacenter is down", you can use inhibit rules to prevent all other alerts from notifying.

What did you expect to see? An explanation of why inhibit rules use a target and source instead of just a basic target matching pattern.

I would also expect to see more expressive answers on Stack Overflow and in the mailing list.

Examples:

  • https://groups.google.com/g/prometheus-users/c/i-jOoQYI_mk/m/nPqACn0_BgAJ (just read the docs, no helpful explainer)
  • https://groups.google.com/g/prometheus-users/c/CgJv15_Oz6c/m/IDAl8D-bBAAJ (no answers for 5+ months)
  • https://stackoverflow.com/questions/57305247/how-to-inhibit-alerts-outside-business-hours-with-prometheus-alertmanager (no explanation of what the point of source and target is. Doesn't use equal and is using deprecated fields.
  • https://groups.google.com/g/prometheus-users/c/xH7jTZdoEV8/m/CL51VmFrAgAJ (this one is actually kind of good because the expert explains that inhibit rules are meant to handle complex scenarios)
  • 👉 finally found someone who can explain the point of inhibit rules https://groups.google.com/g/prometheus-users/c/jPy3uqa83no/m/JqlowJBQAgAJ

Freyert avatar Jul 07 '22 19:07 Freyert

Same here, still no clear answer?

falyoun avatar Jun 14 '23 12:06 falyoun

Same here, need to clearly understand about inhibit rules

AnhQKatalon avatar Oct 29 '23 10:10 AnhQKatalon

It looks like the docs (https://prometheus.io/docs/alerting/latest/configuration/#inhibit_rule) have been updated since this issue was created? It seems to explain both targets and sources?

grobinson-grafana avatar Nov 19 '23 13:11 grobinson-grafana