prometheus icon indicating copy to clipboard operation
prometheus copied to clipboard

Prometheus Operator injects a null receiver in Alertmanager configuration when using alertmanagerConfigSelector

Open MariusRenta opened this issue 4 months ago • 3 comments

What did you do?

I configured two separate Alertmanager configurations (kind: AlertmanagerConfig) and referenced them in my primary Alertmanager (kind: Alertmanager) configuration using alertmanagerConfigSelector with specific labels:

alertmanagerConfigSelector: matchLabels: alertmanagerConfig: monitoring

Each AlertmanagerConfig has receivers specified without any null receiver and with continue set to false.

What did you expect to see?

I expected that each alert would be sent only to the relevant Alertmanager configuration’s receivers and not duplicated. Specifically, alerts should only go to the intended receivers, and there should be no extra, undefined null receiver with continue: true affecting the routing.

What did you see instead? Under which circumstances?

Despite the configuration, the Prometheus Operator injects a null receiver in the global Alertmanager configuration, setting continue: true for each AlertmanagerConfig. This results in duplicated alerts being routed to both configurations, even though the continue option is explicitly set to false in my configurations. This behavior occurs every time I apply the configurations with alertmanagerConfigSelector.

System information

No response

Prometheus version

2.54.1

Prometheus configuration file

No response

Alertmanager version

0.27.0

Alertmanager configuration file

apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  annotations:
  labels:
    app: kube-prometheus-stack-alertmanager
    app.kubernetes.io/instance: prom
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: kube-prometheus-stack
    app.kubernetes.io/version: 65.0.0
    chart: kube-prometheus-stack-65.0.0
    heritage: Helm
    release: prom
  name: prom-kube-prometheus-stack-alertmanager
  namespace: monitoring
spec:
  alertmanagerConfigNamespaceSelector: {}
  alertmanagerConfigSelector:
    matchLabels:
      alertmanagerConfig: monitoring
  alertmanagerConfiguration: {}
  automountServiceAccountToken: true
  externalUrl: http://prom-kube-prometheus-stack-alertmanager.monitoring:9093
  image: quay.io/prometheus/alertmanager:v0.27.0
  listenLocal: false
  logFormat: logfmt
  logLevel: info
  paused: false
  portName: http-web
  replicas: 1
  retention: 120h
  routePrefix: /
  securityContext:
    fsGroup: 2000
    runAsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
    seccompProfile:
      type: RuntimeDefault
  serviceAccountName: prom-kube-prometheus-stack-alertmanager
  version: v0.27.0

Logs

No response

MariusRenta avatar Oct 11 '24 14:10 MariusRenta