PagerDuty configuration: unable to set severity field using template

[Shanti-G]

What did you do? I am trying to set the severity using templated label with below pagerduty configuration in alertmanager:

  pagerduty_configs:
  - send_resolved: true
    routing_key: xxx
    description: |-
      test-cluster.{{ range .Alerts }}{{ .Annotations.summary }}.
      {{ end }}
    severity: '{{ .Labels.pdseverity }}'

I am setting .Labels.pdseverity on an alert in Prometheus.

What did you expect to see? The alertmanager should be able to trigger a PD alert with its severity set based on the templated label.

What did you see instead? Under which circumstances? PagerDuty v2 API allows severity of alert to be set, default being error. With the above configuration, though an alert is in firing state in Prometheus, I don't get the triggered alert in PD.

However, when I directly set the value, like below, I get to see the incident in PD with the given severity, i.e. warning:

  pagerduty_configs:
  - send_resolved: true
    routing_key: xxx
    description: |-
      test-cluster.{{ range .Alerts }}{{ .Annotations.summary }}.
      {{ end }}
    severity: 'warning'

Would like to know how to set the severity field with a templated value. Thanks in advance.

Environment

• Alertmanager version: alertmanager, version 0.21.0 (branch: HEAD, revision: 4c6c03ebfe21009c546e4d1e9b92c371d67c021d) build user: root@dee35927357f build date: 20200617-08:54:02 go version: go1.14.4

• Prometheus version: prometheus, version 2.27.1 (branch: HEAD, revision: db7f0bcec27bd8aeebad6b08ac849516efa9ae02) build user: root@fd804fbd4f25 build date: 20210518-14:17:54 go version: go1.16.4 platform: linux/amd64

• Logs: No error logs from alertmanager to indicate configuration error.

[sbrass]

You made the mistake of using .Labels, which is part of the Alert structure.

You may want to use .CommonLabels at that point? Check the documentation https://prometheus.io/docs/alerting/latest/notifications/.

[Shanti-G]

Thanks @sbrass for getting back on this. We have the pdseverity defined for each alert at Alert level. Would you be having any suggestion on getting this alert level data here? Thanks.

[sbrass]

The answer depends on your configuration, for example chosen routes and your application case.

I would try to route explicitly by your pdseverity label, i.e., all alerts send to a receiver will share a common label, pdseverity, which you then can access by .CommonLabels.pdseverity.

Alternatively, if you group alerts by cluster, service and/or host and have multiple alerts with possible differing pdseverity, you will want to choose the highest pdseverity to apply as payload severity? I would expect Go Template to be able to handle such a case, where you want to have the highest severity printed. However, I can not come up with a quick example for such an application.

[mike2194]

isn't the severity not even passed to PD? seems like Severity should be mentioned in notifiers.go

PR#3739