Is it possible to de-duplicate alerts differentiated by receivers in the UI?

[mmiller1]

Our alertmanager instances will route alerts to any number of receivers based on labels attached to the alert, this can often result in alerts being duplicated in Karma's UI multiple times. In the documentation, I see that I can exclude, or keep alerts routed to specific receivers, however, I don't know ahead of time which combination(s) of receivers will actually receive the alerts, so using this isn't feasible. Is it possible to simply de-duplicate the alerts, ignoring the value in receiver? I tried adding "@\receiver" to the labels.strip field, however, this doesn't work given it's not really a label. Any suggestions are appreciated.

[filippog]

I can echo the request, and would also like to see this feature (or suggestions on how to do achieve it if already possible!)

[filippog]

@prymitive I'm interested in working on this as we have multiple receivers and the dashboard is getting confusing pretty fast now. I'd like some guidance on what do you think is the best way forward, if you don't mind? thank you!

[MolinaRomain]

Update on this one, sometimes being too specific and trying to play around alertmanager's limits can lead to duplicated alerts Any advice would be appreciated ! Thank you !

[filippog]

Here's what I came up with as a possible solution:

• Extend receivers configuration with sth like strip_if_duplicated, it is a list of receiver names
• Semantics are similar to strip although during alertmanager.DedupAlerts() the following will happen:
  • Build a map of labelFP -> [receivers] for all unique alertgroups
  • While building the final dedupedGroups, if the receiver is part of strip_if_duplicated and the current alert's labelFP map entry has more than one receiver, skip the alert

This is similar to actions performed by transform package functions, though in a "stateful" way since we also need to keep track of labelFP -> receivers association to know if an alert should be skipped.

What do folks think?

[craized]

@filippog I appreciate you trying to move forward on this. My 2c below.

I think the most useful functionality here is merging alerts for all @receivers into a single alert with multiple @receiver labels. This provides the clear picture of what alerts are firing and need silencing without removing visibility into alert routing. I can imagine the scenario where only deduping certain @receivers is desired but I'm thinking that's a less common case.

For configuration I think verbiage using a merge keyword is more appropriate if the desired behavior is indeed to unify into a single aggregate alert.

receivers:
  merge_alerts: [truthy]

[filippog]

Thank you for your ideas and input @craized ! I agree that merging receivers is more intuitive, though as far as I can see it would be a more invasive change: internal.models.AlertGroup right now has a single receiver. I'm sure it is doable to change karma to turn that into a list of receivers; though I'd like @prymitive input too!