Receiving alert messages with Feishu is very slow

[Bennu-Li]

Using curl to access the /api/v2/alerts interface to send notifications to Feishu, it takes about 40 seconds. I also tried using prometheus alertmanager as the sender, it's also slow. Is this normal and what can be done to improve it?

[DesistDaydream]

This is a normal phenomenon. NM is asynchronous. When an alarm is received, it will be pushed to the Provider, and then the Dispatcher will pull the alarm from the Provider and perform notification operations in multiple stages.

[benjaminhuo]

@wanjunlei Do we have a wait period before actually sending the alert to feishu ?

[wanjunlei]

The following situations can cause delays.

First, the Notification Manager handles the incoming alerts with a batch processing mechanism. The incoming alerts will not be processed immediately but will be cached for a period of time, the default is 1 minute. So this will cause a delay of no more than 1 minute. You can change this time follow this.

kubectl edit nm notification-manager

And change the spec.batchMaxWait.

Second, if you send notifications to Feishu users or departments, it will have an indeterminate delay. The reason is that sending notifications to the user is an asynchronous operation, and the Feishu server determines this delay.