prometheus
prometheus copied to clipboard
Prometheus Operator injects a null receiver in Alertmanager configuration when using alertmanagerConfigSelector
What did you do?
I configured two separate Alertmanager configurations (kind: AlertmanagerConfig) and referenced them in my primary Alertmanager (kind: Alertmanager) configuration using alertmanagerConfigSelector with specific labels:
alertmanagerConfigSelector: matchLabels: alertmanagerConfig: monitoring
Each AlertmanagerConfig has receivers specified without any null receiver and with continue set to false.
What did you expect to see?
I expected that each alert would be sent only to the relevant Alertmanager configuration’s receivers and not duplicated. Specifically, alerts should only go to the intended receivers, and there should be no extra, undefined null receiver with continue: true affecting the routing.
What did you see instead? Under which circumstances?
Despite the configuration, the Prometheus Operator injects a null receiver in the global Alertmanager configuration, setting continue: true for each AlertmanagerConfig. This results in duplicated alerts being routed to both configurations, even though the continue option is explicitly set to false in my configurations. This behavior occurs every time I apply the configurations with alertmanagerConfigSelector.
System information
No response
Prometheus version
2.54.1
Prometheus configuration file
No response
Alertmanager version
0.27.0
Alertmanager configuration file
apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
annotations:
labels:
app: kube-prometheus-stack-alertmanager
app.kubernetes.io/instance: prom
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/part-of: kube-prometheus-stack
app.kubernetes.io/version: 65.0.0
chart: kube-prometheus-stack-65.0.0
heritage: Helm
release: prom
name: prom-kube-prometheus-stack-alertmanager
namespace: monitoring
spec:
alertmanagerConfigNamespaceSelector: {}
alertmanagerConfigSelector:
matchLabels:
alertmanagerConfig: monitoring
alertmanagerConfiguration: {}
automountServiceAccountToken: true
externalUrl: http://prom-kube-prometheus-stack-alertmanager.monitoring:9093
image: quay.io/prometheus/alertmanager:v0.27.0
listenLocal: false
logFormat: logfmt
logLevel: info
paused: false
portName: http-web
replicas: 1
retention: 120h
routePrefix: /
securityContext:
fsGroup: 2000
runAsGroup: 2000
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
serviceAccountName: prom-kube-prometheus-stack-alertmanager
version: v0.27.0
Logs
No response