Prometheus Operator injects a null receiver in Alertmanager configuration when using alertmanagerConfigSelector

[MariusRenta]

What did you do?

I configured two separate Alertmanager configurations (kind: AlertmanagerConfig) and referenced them in my primary Alertmanager (kind: Alertmanager) configuration using alertmanagerConfigSelector with specific labels:

alertmanagerConfigSelector: matchLabels: alertmanagerConfig: monitoring

Each AlertmanagerConfig has receivers specified without any null receiver and with continue set to false.

What did you expect to see?

I expected that each alert would be sent only to the relevant Alertmanager configuration’s receivers and not duplicated. Specifically, alerts should only go to the intended receivers, and there should be no extra, undefined null receiver with continue: true affecting the routing.

What did you see instead? Under which circumstances?

Despite the configuration, the Prometheus Operator injects a null receiver in the global Alertmanager configuration, setting continue: true for each AlertmanagerConfig. This results in duplicated alerts being routed to both configurations, even though the continue option is explicitly set to false in my configurations. This behavior occurs every time I apply the configurations with alertmanagerConfigSelector.

System information

No response

Prometheus version

2.54.1

Prometheus configuration file

No response

Alertmanager version

0.27.0

Alertmanager configuration file

apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  annotations:
  labels:
    app: kube-prometheus-stack-alertmanager
    app.kubernetes.io/instance: prom
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: kube-prometheus-stack
    app.kubernetes.io/version: 65.0.0
    chart: kube-prometheus-stack-65.0.0
    heritage: Helm
    release: prom
  name: prom-kube-prometheus-stack-alertmanager
  namespace: monitoring
spec:
  alertmanagerConfigNamespaceSelector: {}
  alertmanagerConfigSelector:
    matchLabels:
      alertmanagerConfig: monitoring
  alertmanagerConfiguration: {}
  automountServiceAccountToken: true
  externalUrl: http://prom-kube-prometheus-stack-alertmanager.monitoring:9093
  image: quay.io/prometheus/alertmanager:v0.27.0
  listenLocal: false
  logFormat: logfmt
  logLevel: info
  paused: false
  portName: http-web
  replicas: 1
  retention: 120h
  routePrefix: /
  securityContext:
    fsGroup: 2000
    runAsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
    seccompProfile:
      type: RuntimeDefault
  serviceAccountName: prom-kube-prometheus-stack-alertmanager
  version: v0.27.0

Logs

No response

[MariusRenta]

apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: labels: app: kube-prometheus-stack-prometheus app.kubernetes.io/instance: prom app.kubernetes.io/managed-by: Helm app.kubernetes.io/part-of: kube-prometheus-stack app.kubernetes.io/version: 65.0.0 chart: kube-prometheus-stack-65.0.0 heritage: Helm release: prom name: prom-kube-prometheus-stack-prometheus namespace: monitoring spec: alerting: alertmanagers: - apiVersion: v2 name: prom-kube-prometheus-stack-alertmanager namespace: monitoring pathPrefix: / port: http-web enableAdminAPI: false enableFeatures: - exemplar-storage - remote-write-receiver - native-histograms evaluationInterval: 30s externalUrl: http://prom-kube-prometheus-stack-prometheus.monitoring:9090 hostNetwork: false image: quay.io/prometheus/prometheus:v2.54.1 listenLocal: false logFormat: logfmt logLevel: info paused: false podMonitorNamespaceSelector: {} podMonitorSelector: matchLabels: release: prom portName: http-web probeNamespaceSelector: {} probeSelector: matchLabels: release: prom replicas: 1 retention: 2d routePrefix: / ruleNamespaceSelector: {} ruleSelector: matchLabels: release: prom scrapeConfigNamespaceSelector: {} scrapeConfigSelector: {} scrapeInterval: 30s securityContext: fsGroup: 2000 runAsGroup: 2000 runAsNonRoot: true runAsUser: 1000 seccompProfile: type: RuntimeDefault serviceAccountName: prom-kube-prometheus-stack-prometheus serviceMonitorNamespaceSelector: {} serviceMonitorSelector: {} shards: 1 storage: volumeClaimTemplate: metadata: name: prom-data-prometheus-prom-kube-prometheus-stack-prometheus-0 spec: accessModes: - ReadWriteOnce resources: requests: storage: 8Gi storageClassName: local-storage tsdb: outOfOrderTimeWindow: 0s version: v2.54.1 walCompression: true

[MariusRenta]

apiVersion: monitoring.coreos.com/v1alpha1 kind: AlertmanagerConfig metadata: labels: alertmanagerConfig: monitoring namespace: monitoring name: dev-alertmanagerconfig spec: inhibitRules: - equal: - alertname sourceMatch: - name: severity value: critical targetMatch: - name: severity value: warning - equal: - alertname sourceMatch: - name: severity value: critical targetMatch: - name: severity value: info - equal: - alertname sourceMatch: - name: severity value: warning targetMatch: - name: severity value: info receivers: - emailConfigs: - from: *** requireTLS: false sendResolved: true smarthost: *** to: *** name: team-dev-mails route: continue: false groupBy: - alertname groupInterval: 2m groupWait: 2m receiver: team-dev-mails repeatInterval: 1h routes: - continue: false matchers: - matchType: '=~' name: pod value: >- .(pod1|pod2). receiver: team-dev-mails

[MariusRenta]

apiVersion: monitoring.coreos.com/v1alpha1 kind: AlertmanagerConfig metadata: labels: alertmanagerConfig: monitoring name: global-alertmanagerconfig namespace: monitoring spec: inhibitRules: - equal: - alertname sourceMatch: - name: severity value: critical targetMatch: - name: severity value: warning - equal: - alertname sourceMatch: - name: severity value: critical targetMatch: - name: severity value: info - equal: - alertname sourceMatch: - name: severity value: warning targetMatch: - name: severity value: info receivers: - name: slack-notifications slackConfigs: - apiURL: key: alertmanager.yaml name: slack.yaml channel: '#***' sendResolved: true text: |- Mediu: **** {{ range .Alerts }} Alert: {{ .Annotations.summary }} Descriere: {{ .Annotations.description }} Severitate: {{ .Labels.severity }} {{ end }} title: >- [{{ .Status }}] {{ .GroupLabels.alertname }} ({{ .CommonLabels.severity }}) route: continue: false groupBy: - alertname groupInterval: 2m groupWait: 2m receiver: slack-notifications repeatInterval: 1h routes: - continue: false receiver: slack-notifications

[machine424]

For prometheus-operator related questions, you may want to try https://kubernetes.slack.com/archives/CFFDS2Z7F or https://github.com/prometheus-operator/prometheus-operator/issues (maybe related to https://github.com/prometheus-operator/prometheus-operator/issues/6003?)