Expose opsgenie alias configuration

[antonio-tolentino]

This change will help Opsgenie to deduplicate Alerts when it is necessary to send the same alert to multiple teams and each team is using its own integration instead of a global integration.

[antonio-tolentino]

Hi @simonpasquier, Thank you for comment here.

I have a specific situation here that I need to send the same alert to 2 different teams in Opsgenie. When Alertmanager send alerts to Opsgenie, it uses "group_by" config field to compose the alias. This way Opsgenie will receive 2 occurrences for the same alert because the alias generated is the same and just 1 of the teams will receive this alert. To deduplicate this in Opsgenie would be necessary to include "team name" for example which is not current possible.

If you need more information please let me know.

[kbudde]

Hey 🖖,

nice to see more people working with opsgenie and facing the same issues. I opened #3640 to address the same issue, this MR is trying to solve :)

@antonio-tolentino Maybe you can have a look at @3640. I think only allowing the user to add a prefix the alias is the better approach. Alertmanager is already grouping alerts. The alias is following the alertmanager grouping -> You will have the same alerts in Opsgenie. If you allow chaning the labels for grouping for opsgenie, you will get a different grouping of alerts there.

@simonpasquier I tried to explain the shenanigans of opsgenie in #3639. The alias in opsgenie is used to dedpulicate incoming alerts. Unfortunately, if an alert is coming in it is even deduplicated across different teams. Which leads to undefined behaviour (open alert for team a, but it close for team b). Therefore each team should ensure, that they have unique "alias" values.

Note there is also an open MR to fix missing updates, if the alert is resolved: #3430.

Hope this helps

[antonio-tolentino]

Hi @kbudde,

I do believe that if we combine group_by labels and alias_prefix we can definitely deduplicate alerts in Opsgenie. Please, let's go for this and if you need anything from my side let me know.

Thanks a lot!

[kbudde]

Hey, this is what I tried to tell you. You can already do this with the prefix itself. No need to change the grouping labels in the alias. This will lead to surprises.

You have one integration per team as we have it as well. Setting an alias as prefix (e.g. teamA) will Ensure unique alias between teams (no dedup cross team👍) AND keep the same grouping as in alertmanager. If you have a different grouping (alias) in opsgenie than in alertmanager, you will run into different amount of open alerts in both systems. So the only allowed change to the alias should be to extend it (fixed prefix or suffix) but not change the grouping labels IMHO.

You can set up a prefix in opsgenie per integration ( needs some license). This works quite nice already. We have it running since a long time like this. It's only an issue if alertmanager tries to update an alert as the prefix is not applied on server side in for update operations.

[antonio-tolentino]

If you are thinking in only setup alias per team in Opsgenie, this won't work for us. we currently have more than 150 K8s clusters and many teams per cluster and these figures will increase. The alias(group_by labels + alias_prefix) has to be generated by Alertmanager, in order to get them deduplicated, opened and closed automatically in Opsgenie side. This will guarantee team A does not close alert from team B and vice-versa and not less important I don't have to setup alias team by team. We do need #3640 code merged.