Feature Request: Allow Relabeling of Alerts

[siegenthalerroger]

In our alerting stack we use Prometheus AlertManager and use a defined list of labels to correctly route any Alerts that are pushed to AM. The alerts sent by the notification controller to AM however use a different list of labels (or even more precisely different values than we expect). It would be beneficial to have way to relabel the alerts before they are sent to other systems.

I see two ways of our issue being properly solved. The easy version would be to allow renaming of labels before they are sent (e.g. namespace -> vendor). This does not address the issue of changing alert values which is considerably harder and I would consider that out of scope.

[antoineozenne]

We have the same problem. We need to route the notification to the right team based on a specific label. Maybe it could be possible to allow adding custom label in the Alert?

[Alan01252]

I knocked up a proof of concept around this and it's relatively trivial to implement.

The sticking point is how to get the configuration into the provider. At the moment providers don't allow you to have a bespoke configuration passed in for that provider.

[Alan01252]

@stefanprodan do you have any thoughts on the above? The crux of the problem is we need some way to pass bespoke configuration to a provider.

My thoughts are that the provider could optionally take a ConfigurationRef containing ( optional ) bespoke configuration that could be read by the provider when instantiated.

I'd like to work on implementing this if you feel there's merit to the approach.

[stefanprodan]

We copy the event metadata to the alertmanager labels, based on this we could inject the Alert.metadata.labels into the event before it gets sent to the provider. While this will allow setting custom labels it will not allow relabelling such as namespace -> vendor.

[Alan01252]

Would that mean the flux project isn't open to ConfigurationRef style solution ( and that's a perfectly acceptable response ) to satisfy the relabelling feature?

My ( and I never would expect this pull request to be merged/consider to be merged ) brute force proof of concept implementation is here for those interested https://github.com/fluxcd/notification-controller/compare/main...Alan01252:notification-controller:feature/alert-relabelling

[siegenthalerroger]

@stefanprodan adding custom labels would already be useful, however relabelling would definitely be useful as well.

[Alan01252]

We copy the event metadata to the alertmanager labels, based on this we could inject the Alert.metadata.labels into the event before it gets sent to the provider. While this will allow setting custom labels it will not allow relabelling such as namespace -> vendor.

Something as simple as https://github.com/Alan01252/notification-controller/commit/b23dc546bfd5345f2d3736cf7c7e1fa482cfc7d9 would work I guess. Need to be cautious about labels potentially being overridden by the alert labels.