alertmanager icon indicating copy to clipboard operation
alertmanager copied to clipboard

Inhibits should be per-route rather than global

Open brian-brazil opened this issue 10 years ago • 5 comments

Inhibits have the potential for accidentally affecting far more alerts than is desired, particularly in environments where more than one team is sharing the alertmanager (and we'll ultimately encourage one set of alertmanagers per company).

Accordingly it'll be safer for inhibitions to be per-route, and likely also easier to work with.

brian-brazil avatar Dec 06 '15 18:12 brian-brazil

I don't think this will work out well.

It is still necessary to create global sciences – e.g. if I perform a maintenance that I know will cause phony alerts; I can't track down which routes that would go down.

An alert may go to more than one route, how would that be handled?

And finally, "silences per route" introduces yet another level of reasoning and makes silences even harder to understand. This will encourage simply silencing everything for a route, because "it's only our own stuff".

We should make it easier to understand and design silences of the correct scope, not artificially limit the scope along some dimension.

matthiasr avatar Dec 07 '15 09:12 matthiasr

It is still necessary to create global sciences – e.g. if I perform a maintenance that I know will cause phony alerts; I can't track down which routes that would go down.

Inhibits are about firing alerts preventing other firing alerts, I'm not seeing how silences come into it.

I think you're confusing inhibits and silences.

brian-brazil avatar Dec 07 '15 09:12 brian-brazil

Indeed I am. You see, too many layers! No opinion either way from me then, since I haven't worked with inhibits too much yet.

matthiasr avatar Dec 07 '15 09:12 matthiasr

:+1: It confused me to see inhibit_rules being a global setting.

grobie avatar Jan 05 '16 19:01 grobie

+1

oleg-glushak avatar Dec 14 '17 10:12 oleg-glushak