alertmanager icon indicating copy to clipboard operation
alertmanager copied to clipboard

Feature Request: Resend active alerts after silencer is expired.

Open hatosbarbosa opened this issue 2 years ago • 3 comments

Hey there! I've been using alertmanager for a couple years, and it's been a great tool, thanks for the great job you guys have done!

From what I've experienced, when a silencer is created it won't let the firing alerts to be sent to the receivers. But when the silencer is expired, the firing alerts aren't sent right away, it respects the repeat_interval value in the alertmanager.yml . It makes sense to me, but: Is there a way to force the firing alerts to be sent to the receivers right after the silencer is expired?

If there's a way to do it already, could someone explain to me how it's done? And if there's not, can it be developed?

I'm not a go dev, but I could try to develop a feature like this if someone is able just to guide me for a couple minutes through the project (and the current architecture allows it, of course)

Thanks!

hatosbarbosa avatar Apr 29 '22 13:04 hatosbarbosa

HI @hatosbarbosa,

Thanks for reaching out. This is something that's definitely on my radar, but it's a bit more complex than that. I think that has a first step, we need to get the terminology right - right now we have concepts of muting, silencing, inhibitions, etc which kind of are related to the same thing.

Without agreement on terminology, It's impossible to know whether something like fire waiting for notifications on repeat_interval` if silences expire" is a breaking change or not.

I feel like this is mostly a duplicate of https://github.com/prometheus/alertmanager/issues/2496 and it's best if we keep the conversation focused around it. Feel free to drop your thoughts on that issue and please 👍 the issue as this helps us prioritise a bit better.

gotjosh avatar Apr 29 '22 14:04 gotjosh

@roidelapluie or @simonpasquier can you please close this?

gotjosh avatar Apr 29 '22 14:04 gotjosh

Aside:

it respects the repeat_interval

@hatosbarbosa you mean the group_interval, not the repeat_interval, right?

juliusv avatar May 02 '22 19:05 juliusv