Alertmanager alerts do not show up if neither WARNING nor CRITICAL

[gerdriesselmann]

Nagstamon version: 3.8.0 and earlier

What causes issue ?

We have Prometheus alerts that have a severity of "unreachable". Nagstamon reads this via Alertmanager.

Expected behavior?

The alerts show up as "UNREACHABLE" in Nagstamon and cause an according alert.

This is how I understand the method "map_severity" in the AlertmanagerServer class.

What happens instead?

Alerts are neither shown in status window list of alerts nor does a notification appear.

Any more details?

If I add "unreachable" to the list in the server setting "Map to CRITICAL", the alerts are indeed classified as CRITICAL and do show up.

This IMO shows they have been successfully retrieved and are indeed discarded later on by Nagstamon for some reason.

All filters are unchecked. Notifications are enabled for every status (UNKNOWN to DOWN)

[HenriWahl]

@stearz any idea?

[Noki]

I can confirm that this bug is still present in 3.10.1.

[HenriWahl]

@gerdriesselmann @Noki did it work with ANY version of Nagstamon before?

[gerdriesselmann]

@HenriWahl I unfortunately don't know. We introduced the UNREACHABLE severity in December 2021 when 3.8 was the latest release.

[HenriWahl]

@gerdriesselmann is this fixed for you with a newer testing release?

[gerdriesselmann]

@HenriWahl Unfortunately not :( I tried 3.13-20240108

[HenriWahl]

@stearz do you have any hint here?

[stearz]

I'm afraid this is a limitation in alertmanager as it only supports service objects but the status UNREACHABLE only exists in Nagstamon's Server objects not in the Service objects. So unless there will be a big refactoring of Nagstamon I think the "solution" will be to add the status "unreachable" to the "Map to CRITICAL" like you already did.