alertmanager
alertmanager copied to clipboard
Documentation for inhibit rules is not clear.
What did you do?
I'm trying to use inhibit rules for the first time. After reading the documentation many times I have no idea why there is both a target alert and a source alert. Why is there not just a single target alert that I'm trying to inhibit?
I've reviewed many mailing list conversations, GH issues, and Stack Overflow posts regarding inhibit rules, but no one seems to understand how they work. Answerers present new configuration to try, but never display a deep understanding of the thinking behind inhibit rules themselves.
I think also that alert_relabel_configs are a bit more straightforward to use if you just want to drop alerts with particular labels.
Update: I managed to find a mailing list post that explains inhibit_rules in a way that was easier to grasp for me than the documentation. https://groups.google.com/g/prometheus-users/c/jPy3uqa83no/m/JqlowJBQAgAJ
Inhibit rules specify how one alert can inhibit (~silence) another – say, you have an alert that says "the whole datacenter is down", you can use inhibit rules to prevent all other alerts from notifying.
What did you expect to see?
An explanation of why inhibit rules use a target and source instead of just a basic target matching pattern.
I would also expect to see more expressive answers on Stack Overflow and in the mailing list.
Examples:
- https://groups.google.com/g/prometheus-users/c/i-jOoQYI_mk/m/nPqACn0_BgAJ (just read the docs, no helpful explainer)
- https://groups.google.com/g/prometheus-users/c/CgJv15_Oz6c/m/IDAl8D-bBAAJ (no answers for 5+ months)
- https://stackoverflow.com/questions/57305247/how-to-inhibit-alerts-outside-business-hours-with-prometheus-alertmanager (no explanation of what the point of
sourceandtargetis. Doesn't useequaland is using deprecated fields. - https://groups.google.com/g/prometheus-users/c/xH7jTZdoEV8/m/CL51VmFrAgAJ (this one is actually kind of good because the expert explains that inhibit rules are meant to handle complex scenarios)
- 👉 finally found someone who can explain the point of inhibit rules https://groups.google.com/g/prometheus-users/c/jPy3uqa83no/m/JqlowJBQAgAJ
Same here, still no clear answer?
Same here, need to clearly understand about inhibit rules
It looks like the docs (https://prometheus.io/docs/alerting/latest/configuration/#inhibit_rule) have been updated since this issue was created? It seems to explain both targets and sources?