[kube-prometheus-stack] Alertmanager SNS config, cannot modify the Subject

[ratkokorlevski]

Describe the bug a clear and concise description of what the bug is.

I have installed "kube-prometheus-stack" for my EKS cluster on AWS. Then I need all alerts to be received on my monitoring SNS topics and create Jira issues from there with a convinient title. But I can't configure the Subject of the message that Alertmanager sends to SNS topic. Currently I am only getting "[FIRING:1]".

This the setup of the helm chart:

resource "helm_release" "kube-prometheus-stack" {
       name       = "kube-prometheus-stack"
       repository = "https://prometheus-community.github.io/helm-charts"
       chart      = "kube-prometheus-stack"
       version    = "48.3.1"
       namespace  = "monitoring"
       values = [
           file("./helm-values/prometheus.yaml"),
           file("./helm-values/alertmanager.yaml"),
           file("./helm-values/grafana.yaml")
       ]
}

So I tried to set up the subject with these alertmanager values:

alertmanager:
  enabled: true
  templateFiles:
    custom_sns_config.tmpl: |-
      {{ define "sns.default.subject" }}{{ .Annotations.description }}{{ end }}       <----HERE
      {{ define "sns.default.message" }}{{ with index .Alerts 0 -}}
                  :chart_with_upwards_trend: *<{{ .GeneratorURL }}|Graph>*
                  {{- if .Annotations.runbook_url }}   :notebook: *<{{ .Annotations.runbook_url }}|Runbook>*{{ end }}
                {{ end }}

                *Alert details*

                {{ range .Alerts -}}
                  *Alert:* {{ .Labels.alertname }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}
                  {{- if .Annotations.summary }}
                  *Summary:* {{ .Annotations.summary }}{{ end }}
                  {{- if .Annotations.description }}
                  *Description:* {{ .Annotations.description }}{{ end }}
                  {{- if .Annotations.message }}
                  *Message:* {{ .Annotations.message }}{{ end }}
                  *Details:*
                  {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
                  {{ end }}
                {{ end }}
  config:
    route:
    ...
      receivers:
        - name: "severity-critical"
          sns_configs:
            - topic_arn: "arn:aws:sns:eu-north-1:ACCOUNT_ID:production-high-priority-topic"
              api_url: "sns.eu-north-1.amazonaws.com"
              subject: {{ .Annotations.description }}{{ end }}     <---------HERE
              sigv4:
                region: "eu-north-1"

I understand that previously these encountered issues in terms of "Subject too long or else" but I should able to configure it if I want. From the values documentation I can see that you can add subject to receivers.sns_configs[0].subject or override he config files by adding template file in alertmanager.templateFiles

What's your helm version?

version.BuildInfo{Version:"v3.12.0", GitCommit:"c9f554d75773799f72ceef38c51210f1842a1dea", GitTreeState:"clean", GoVersion:"go1.20.4"}

What's your kubectl version?

Client Version: v1.24.0 Kustomize Version: v4.5.4 Server Version: v1.28.4-eks-8cb36c9

Which chart?

kube-prometheus-stack

What's the chart version?

48.3.1

What happened?

No response

What you expected to happen?

No response

How to reproduce it?

No response

Enter the changed values of values.yaml?

No response

Enter the command that you execute and failing/misfunctioning.

terraform apply but this is for the whole stack The issue is afterwards of the performance of alertmanager

Anything else we need to know?

No response

[ratkokorlevski]

Can anyone assist with this, how it should be done ? I need to modify the subject or at least send the data as JSON so I can modify the data

@dotdc @zwopir @sathieu @piotrhryszko-img @Stelminator @Bregor @kesor @redondos

[zeritti]

A couple of points:

{{ define "sns.default.subject" }}{{ .Annotations.description }}{{ end }}

This won't work as the Data object does not contain Annotations which is present only in the Alert object [0] and should therefore result in an error. Furthermore, as there might be many matching alerts depending on grouping, which description should be used?

If you are grouping by e.g. alertname, you can use alertname from GroupLabels:

{{ define "sns.default.subject" }}{{ .GroupLabels.alertname }}{{ end }}

In this same case you can also use CommonAnnotations:

{{ define "sns.default.subject" }}{{ .CommonAnnotations.description }}{{ end }}

Otherwise, you can generally extract a field from an alert, e.g.

{{ define "sns.default.subject" }}{{ (index .Alerts 0).Annotations.description }}{{ end }}

subject: {{ .Annotations.description }}{{ end }}

If you define a template, you probably intend to use it:

subject: '{{ template "sns.default.subject" . }}'

Or without a need for a template, e.g.:

subject: '{{ .CommonAnnotations.description }}'

If you name the subject template as you did, you can probably leave the subject out entirely as you use the default subject template name. [1]

When testing and validating a template/configuration, amtool is quite useful.

An example of a template for JSON-formatted notifications. [2]

[0] https://prometheus.io/docs/alerting/latest/notifications/#data [1] https://prometheus.io/docs/alerting/latest/configuration/#sns_config [2] https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-alertmanager-receiver-JSON.html