AlertmanagerConfig matchers not working

[mateoporcar]

I'm trying to generate custom AlertManagerConfig but the matches added in the yaml are not reflected in the alertmanager configuration.

What did you do?

AlertmanagerConfig.yaml

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: config-example
  namespace: cert-checker
  labels:
    alertmanagerConfig: monitoring
spec:
  route:
    groupBy: ['alertname']
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 24h
    receiver: 'email'
    routes:
    - match:
        alertname: cert-checker-service
      receiver: email
  receivers:
  - name: 'email'
    emailConfigs:
    - to: '[email protected]'
      from: '[email protected]'
      requireTLS: false

Also, I tried this way too, with "matchers"

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: config-example
  namespace: cert-checker
  labels:
    alertmanagerConfig: monitoring
spec:
  route:
    groupBy: ['alertname']
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 24h
    receiver: 'email'
    routes:
    - matchers:
        name: alertname
        value: cert-checker-service
      receiver: email
  receivers:
  - name: 'email'
    emailConfigs:
    - to: '[email protected]'
      from: '[email protected]'
      requireTLS: false

When I log in in the pod and see this config:

global:
  resolve_timeout: 5m
  http_config:
    follow_redirects: true
  smtp_hello: localhost
  smtp_require_tls: true
  pagerduty_url: https://events.pagerduty.com/v2/enqueue
  opsgenie_api_url: https://api.opsgenie.com/
  wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
  victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
route:
  receiver: "null"
  group_by:
  - job
  continue: false
  routes:
  - receiver: cert-checker-config-example-email
    group_by:
    - alertname
    matchers:
    - namespace="cert-checker"
    continue: true
    routes:
    - receiver: cert-checker-config-example-email
      continue: false
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 1d
  - receiver: default-alert-kube-state-metrics-email
    group_by:
    - alertname
    matchers:
    - namespace="default"
    continue: true
    routes:
    - receiver: default-alert-kube-state-metrics-email
      continue: false
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 5m
  - receiver: monitoring-alert-kube-prometheus-stack-deployment-email
    group_by:
    - job
    matchers:
    - namespace="monitoring"
    continue: true
    routes:
    - receiver: monitoring-alert-kube-prometheus-stack-deployment-email
      continue: false
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 1h
  - receiver: "null"
    match:
      alertname: Watchdog
    continue: false
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
receivers:
- name: "null"
- name: cert-checker-config-example-email
  email_configs:
  - send_resolved: false
    to: [email protected]
    from: [email protected]
    hello: localhost

    headers:
      From: [email protected]
      Subject: '{{ template "email.default.subject" . }}'
      To: [email protected]
    html: '{{ template "email.default.html" . }}'
    require_tls: false
- name: default-alert-kube-state-metrics-email
  email_configs:
  - send_resolved: false
    to: [email protected]
    from: [email protected]
    hello: localhost
    headers:
      From: [email protected]
      Subject: '{{ template "email.default.subject" . }}'
      To: [email protected]
    html: '{{ template "email.default.html" . }}'
    require_tls: false
- name: monitoring-alert-kube-prometheus-stack-deployment-email
  email_configs:
  - send_resolved: false
    to: [email protected]
    from: [email protected]
    hello: localhost
    headers:
      From: [email protected]
      Subject: '{{ template "email.default.subject" . }}'
      To: [email protected]
    html: '{{ template "email.default.html" . }}'
    require_tls: false
templates:
- /etc/alertmanager/config/*.tmpl

Did you expect to see some different?

I expect to see in config the match "alertname: cert-checker-service".

• Prometheus Operator version: quay.io/prometheus-operator/prometheus-operator:v0.53.1 and quay.io/prometheus-operator/prometheus-operator:v0.56.0

  Insert image tag or Git SHA here

• Kubernetes version information:

  kubectl version

• Kubernetes cluster kind:

  insert how you created your cluster: kops, bootkube, etc.

• Manifests:

insert manifests relevant to the issue

• Prometheus Operator Logs:

insert Prometheus Operator logs relevant to the issue here

Anything else we need to know?:

I added the following config in Alertmanager:

    alertmanagerConfiguration:
      name: global-alertmanager-configuration
    alertmanagerConfigSelector:
      matchLabels:
        alertmanagerConfig: monitoring

[gnadaban]

Hey @mateoporcar , I had the same issue before and it was due to syntax mismatches. Check the logs of the Prometheus operator and Alertmanager, and see if you get errors there - I did, and it helped me identify and fix the issue.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the last 60 days. Thank you for your contributions.

[github-actions[bot]]

This issue was closed because it has not had any activity in the last 120 days. Please reopen if you feel this is still valid.

[StevenBarre]

This worked for me

spec:
  route:
    groupBy:
    - severity
    - namespace
    groupInterval: 5m
    groupWait: 30s
    receiver: Everyone
    repeatInterval: 48h
    routes:
    - receiver: TechLeads
      repeatInterval: 1w
      matchers:
        - name: severity
          value: warning|info
          matchType: =~