dianping
告警延迟分析
背景 微服务>50，告警>2w时，经常出现告警延迟，最多时候达到1h以上
原因分析
SendExcutor每5min拉取一次告警，通过arthas分析，主要是在 m_decoratorManager.generateTitleAndContent 耗时长，发现是其中有一条SQL执行很慢
SELECT al.id,al.domain,al.alert_time,al.category,al.type,al.content,al.metric,al.creation_date FROM alert al WHERE al.alert_time >= ? AND al.alert_time <= ? AND al.category = ? AND al.domain = ? ORDER BY al.alert_time asc
解决策略
增加索引index idx_alert_time_category_domain(alert_time,category,domain);
alter table alert add index idx_alert_time_category_domain(alert_time,category,domain);
建议 在初始化的CatApplication.sql中，补充该索引
麻烦顺便发个pull request，谢谢！
On Aug 15, 2022, at 14:48, romanticmj @.***> wrote:
背景 微服务>50，告警>2w时，经常出现告警延迟，最多时候达到1h以上
原因分析 SendExcutor每5min拉取一次告警，通过arthas分析，主要是在 m_decoratorManager.generateTitleAndContent 耗时长，发现是其中有一条SQL执行很慢 SELECT al.id,al.domain,al.alert_time,al.category,al.type,al.content,al.metric,al.creation_date FROM alert al WHERE al.alert_time >= ? AND al.alert_time <= ? AND al.category = ? AND al.domain = ? ORDER BY al.alert_time asc
解决策略 增加索引index idx_alert_time_category_domain(alert_time,category,domain); alter table alert add index idx_alert_time_category_domain(alert_time,category,domain);
建议 在初始化的CatApplication.sql中，补充该索引
— Reply to this email directly, view it on GitHub https://github.com/dianping/cat/issues/2243, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASQE73KM2TVQSE5SMT6QRDVZHR2BANCNFSM56RIPGQQ. You are receiving this because you are subscribed to this thread.
Merged into master
On Aug 17, 2022, at 10:31, romanticmj @.*** @.***>> wrote:
PR：#2245 https://github.com/dianping/cat/pull/2245 — Reply to this email directly, view it on GitHub https://github.com/dianping/cat/issues/2243#issuecomment-1217387424, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASQE77BRS2M5PA6YTRIE2TVZRFJ3ANCNFSM56RIPGQQ. You are receiving this because you commented.