Alarm repeat even if set repeat_interval='4h'

[jiaxinyang1]

What did you do? that's my alertmanager configure:

route:
  group_by: [ 'msf_cluster', 'alertname', 'msf_service' ]
  group_wait: 1m
  group_interval: 5m
  repeat_interval: 4h
  receiver: msfmanagement-notification
receivers:
- name: msfmanagement-notification
  webhook_configs:
    - url: http://msfmanagement-notification.msfmanagement.svc.cluster.local:80/v1/alerts
inhibit_rules:
- source_match:
    eventLevel: "ERROR"
  target_match:
    eventLevel: "NOTICE"
  equal: ['msf_cluster', 'alertname', 'msf_service']

this is one of my rule:

    - alert: high_cpu_error
      expr: sum (msf_pod_cpu_usage_ms{msf_service_pod_container!="register"} / (msf_pod_cpu_limit_ms{msf_service_pod_container!="register"}>0) ) by (msf_cluster,msf_service,msf_service_pod,msf_service_pod_container)*100 > 95
      for: 3m
      labels:
        eventType: "HIGH_CPU"
        eventLevel: "ERROR"
        producer: "Alertmanager"
      annotations:
        summary: "Service high cpu usage"
        description: {{`"Service {{$labels.msf_service}} instance {{ $labels.msf_service_pod }} container {{ $labels.msf_service_pod_container }} cpu usage more than 95%"`}}

What did you expect to see?

I want to get repeating alarm every 4 hours。 What did you see instead? Under which circumstances? But i get repeating alarm in a short time

Service apm-db-bridge-system-cluster01 instance system-apm-apm-db-bridge-ddd568847-g2mkq container service cpu usage more than 70%      2022-03-21 08:05:35.574153+00

Service apm-db-bridge-system-cluster01 instance system-apm-apm-db-bridge-ddd568847-g2mkq container service cpu usage more than 70%      2022-03-21 08:00:35.572802+00

The interval is almost a group_interval

Environment

• System information:

  insert output of uname -srm here

• Alertmanager version:

alertmanager, version 0.20.0 (branch: HEAD, revision: f74be0400a6243d10bb53812d6fa408ad71ff32d)
  build user:       root@00c3106655f8
  build date:       20191211-14:13:14
  go version:       go1.13.5

• Prometheus version:

prometheus, version 2.18.2 (branch: HEAD, revision: a6600f564e3c483cc820bae6c7a551db701a22b3)
  build user:       root@130a411dd4ff
  build date:       20200609-09:05:58
  go version:       go1.14.4

• Alertmanager configuration file:

insert configuration here

• Prometheus configuration file:

insert configuration here (if relevant to the issue)

• Logs:

insert Prometheus and Alertmanager logs relevant to the issue here

[gotjosh]

HI @jiaxinyang1, thanks for reporting. It is not clear to me if the labels of the group are all are the same.

You've defined your group to be: group_by: [ 'msf_cluster', 'alertname', 'msf_service' ], but in your message:

Service apm-db-bridge-system-cluster01 instance system-apm-apm-db-bridge-ddd568847-g2mkq container service cpu usage more than 70%

I'm not able to tell, which is which? It's important to note that if any of the labels in the group by clause is different, it is treated as a separate group and will trigger the cycle of wait -> interval -> repeat again.

[jiaxinyang1]

thanks. this is my query result

{msf_cluster="cluster01",msf_service="apm-rest-bridge-apmstage-cluster01",msf_service_pod="apm-apm-rest-bridge-68d9bf5f7b-h8mhm",msf_service_pod_container="service"} | 6.908789499999999

Service apm-db-bridge-system-cluster01 instance system-apm-apm-db-bridge-ddd568847-g2mkq container service cpu usage more than 70%

msf_cluster="cluster01" All metrics for this label are unique values msf_service="apm-db-bridge-system-cluster01"

i don't set label 'msf_service_pod' and label 'msf_service_pod_container' in my group_by .will the same tag content be divided into different groups?

[simonpasquier]

Are you running a single Alertmanager instance or several? Have you enabled the debug log level and checked the logs? If not, please do so and share the logs if you can't find anything.

[jiaxinyang1]

thank you . @simonpasquier

yes. i'am running a single alertmanager .only one Endpoint address in my prometheus Runtime Information i enabled the debug log . i change the alertmanager config reduced repeat_interval to test .

global:
  resolve_timeout: 5m
  http_config: {}
  smtp_hello: localhost
  smtp_require_tls: true
  pagerduty_url: https://events.pagerduty.com/v2/enqueue
  hipchat_api_url: https://api.hipchat.com/
  opsgenie_api_url: https://api.opsgenie.com/
  wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
  victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
route:
  receiver: msfmanagement-notification
  group_by:
  - msf_cluster
  - alertname
  - msf_service
  group_wait: 1m
  group_interval: 5m
  repeat_interval: 20m
inhibit_rules:
- source_match:
    eventLevel: ERROR
  target_match:
    eventLevel: NOTICE
  equal:
  - msf_cluster
  - alertname
  - msf_service
receivers:
- name: msfmanagement-notification
  webhook_configs:
  - send_resolved: false
    http_config: {}
    url: http://msfmanagement-notification.msfmanagement.svc.cluster.local:80/v1/alerts
templates: []

I choose a rule to observe

alert: high_mem_critical
expr: sum
  by(msf_cluster, msf_service, msf_service_pod, msf_service_pod_container) (msf_pod_memory_usage_Mb{msf_service_pod_container!="register"}
  / (msf_pod_memory_limit_Mb{msf_service_pod_container!="register"} > 0))
  * 100 > 95
for: 3m
labels:
  eventLevel: CRITICAL
  eventType: HIGH_MEM
  producer: Alertmanager
annotations:
  description: Service {{$labels.msf_service}} instance {{ $labels.msf_service_pod
    }} container {{ $labels.msf_service_pod_container }} memory usage more than 95%
    ,values is {{$value}}
  summary: Service high mem usage

this is the alertmanager log

level=debug ts=2022-03-25T02:23:21.777Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T02:24:21.777Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T02:25:21.776Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T02:27:21.788Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T02:29:21.778Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T02:29:21.782Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T02:34:21.778Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][resolved]]
level=debug ts=2022-03-25T02:34:21.783Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T02:36:21.780Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T02:37:21.772Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T02:39:21.779Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][resolved]]
level=debug ts=2022-03-25T02:41:21.786Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T02:42:21.786Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T02:43:21.777Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T02:44:21.777Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T02:47:21.787Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][resolved]]
level=debug ts=2022-03-25T02:51:21.784Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T02:52:21.785Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T02:53:21.783Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T02:55:21.778Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T02:57:21.785Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][resolved]]
level=debug ts=2022-03-25T02:59:21.783Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T03:00:21.784Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T03:01:21.784Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T03:05:21.781Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T03:05:21.785Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T03:07:21.788Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T03:09:21.788Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T03:10:21.785Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T03:11:21.788Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T03:15:21.781Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T03:15:21.786Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T03:16:21.785Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T03:20:21.785Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T03:20:21.787Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T03:21:21.787Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T03:25:21.787Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][resolved]]
level=debug ts=2022-03-25T03:26:21.803Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T03:27:21.804Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T03:28:21.790Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T03:30:21.839Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T03:32:21.804Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][resolved]]
level=debug ts=2022-03-25T03:34:23.703Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T03:35:21.801Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T03:35:21.801Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][resolved]]
level=debug ts=2022-03-25T03:44:21.801Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T03:45:21.802Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T03:46:21.802Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T03:48:21.798Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T03:49:21.802Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T03:50:21.802Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][resolved]]
level=debug ts=2022-03-25T04:06:21.765Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T04:07:21.767Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T04:08:21.764Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T04:10:21.773Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T04:12:21.765Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T04:12:21.768Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T04:13:21.757Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T04:15:21.785Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T04:17:21.768Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][resolved]]
level=debug ts=2022-03-25T04:17:21.769Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T04:22:21.769Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][resolved]]
level=debug ts=2022-03-25T04:32:21.784Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T04:33:21.785Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T04:34:21.776Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T04:36:21.778Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T04:38:21.785Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][resolved]]
level=debug ts=2022-03-25T04:40:21.783Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T04:41:21.784Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T04:42:21.786Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T04:44:21.773Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T04:46:21.784Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][resolved]]
level=debug ts=2022-03-25T04:49:21.775Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T04:50:21.776Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T04:51:21.774Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T04:53:21.783Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T04:55:21.776Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][resolved]]
level=debug ts=2022-03-25T04:57:21.795Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T04:58:21.796Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T04:59:21.788Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:01:21.779Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:03:21.783Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:03:21.796Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T05:05:21.789Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:07:21.785Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:08:21.796Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T05:09:21.792Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T05:13:21.797Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][resolved]]
level=debug ts=2022-03-25T05:14:21.782Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:15:21.782Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T05:16:21.791Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:18:21.785Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:20:21.783Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T05:20:21.783Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:22:21.798Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:24:21.787Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:25:21.783Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T05:26:21.791Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:28:21.794Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:30:21.784Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T05:30:21.785Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:32:23.711Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:34:21.790Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:35:21.784Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T05:36:21.794Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:38:21.796Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:40:21.785Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T05:40:21.802Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:42:21.807Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:44:21.792Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T05:45:21.786Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][resolved]]
level=debug ts=2022-03-25T05:54:21.812Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:55:21.813Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T05:56:21.804Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T05:58:21.804Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T06:00:21.813Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T06:00:21.818Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T06:02:21.798Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T06:04:21.792Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T06:05:21.814Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T06:06:21.765Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T06:10:21.769Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T06:10:21.814Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T06:12:21.772Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T06:14:21.781Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T06:15:21.815Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T06:16:27.909Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T06:18:21.768Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T06:20:21.778Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T06:20:21.815Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][resolved]]
level=debug ts=2022-03-25T06:24:21.769Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T06:25:21.770Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T06:25:21.771Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T06:30:21.770Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][resolved]]
level=debug ts=2022-03-25T06:30:21.773Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T06:32:21.775Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T06:34:21.768Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][active]
level=debug ts=2022-03-25T06:35:21.770Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][active]]
level=debug ts=2022-03-25T06:35:23.804Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=high_mem_critical[05d39fd][resolved]
level=debug ts=2022-03-25T06:40:21.772Z caller=dispatch.go:465 component=dispatcher aggrGroup="{}:{alertname=\"high_mem_critical\", msf_cluster=\"cluster01\", msf_service=\"apm-mes-system-cluster01\"}" msg=flushing alerts=[high_mem_critical[05d39fd][resolved]]

I get two alerts at those time:

"apm-mes-system-cluster01"	"{instance=system-apm-apm-mes-75fffc9dc5-fsqtm,msf_service_pod_container=service}"	"2022-03-25 02:24:07.820329+00"
"apm-mes-system-cluster01"	"{instance=system-apm-apm-mes-75fffc9dc5-fsqtm,msf_service_pod_container=service}"	"2022-03-25 02:42:07.838205+00"
"apm-mes-system-cluster01"	"{instance=system-apm-apm-mes-75fffc9dc5-fsqtm,msf_service_pod_container=service}"	"2022-03-25 02:52:07.8077+00"
"apm-mes-system-cluster01"	"{instance=system-apm-apm-mes-75fffc9dc5-fsqtm,msf_service_pod_container=service}"	"2022-03-25 03:00:07.810456+00"
"apm-mes-system-cluster01"	"{instance=system-apm-apm-mes-75fffc9dc5-fsqtm,msf_service_pod_container=service}"	"2022-03-25 03:27:07.813749+00"
"apm-mes-system-cluster01"	"{instance=system-apm-apm-mes-75fffc9dc5-fsqtm,msf_service_pod_container=service}"	"2022-03-25 03:45:07.811464+00"
"apm-mes-system-cluster01"	"{instance=system-apm-apm-mes-75fffc9dc5-fsqtm,msf_service_pod_container=service}"	"2022-03-25 04:07:07.7669+00"
"apm-mes-system-cluster01"	"{instance=system-apm-apm-mes-75fffc9dc5-fsqtm,msf_service_pod_container=service}"	"2022-03-25 04:33:07.772855+00"
"apm-mes-system-cluster01"	"{instance=system-apm-apm-mes-75fffc9dc5-fsqtm,msf_service_pod_container=service}"	"2022-03-25 04:50:07.74466+00"
"apm-mes-system-cluster01"	"{instance=system-apm-apm-mes-75fffc9dc5-fsqtm,msf_service_pod_container=service}"	"2022-03-25 04:58:07.763306+00"
"apm-mes-system-cluster01"	"{instance=system-apm-apm-mes-75fffc9dc5-fsqtm,msf_service_pod_container=service}"	"2022-03-25 04:41:07.799701+00"
"apm-mes-system-cluster01"	"{instance=system-apm-apm-mes-75fffc9dc5-fsqtm,msf_service_pod_container=service}"	"2022-03-25 05:15:07.759754+00"
"apm-mes-system-cluster01"	"{instance=system-apm-apm-mes-75fffc9dc5-fsqtm,msf_service_pod_container=service}"	"2022-03-25 05:40:07.731338+00"
"apm-mes-system-cluster01"	"{instance=system-apm-apm-mes-75fffc9dc5-fsqtm,msf_service_pod_container=service}"	"2022-03-25 05:55:07.749747+00"
"apm-mes-system-cluster01"	"{instance=system-apm-apm-mes-75fffc9dc5-fsqtm,msf_service_pod_container=service}"	"2022-03-25 06:25:07.691202+00"
"apm-mes-system-cluster01"	"{instance=system-apm-apm-mes-75fffc9dc5-fsqtm,msf_service_pod_container=service}"	"2022-03-25 06:35:07.692441+00"