Sentinel icon indicating copy to clipboard operation
Sentinel copied to clipboard

MetricSearcher#findOffset eating 5%~15% CPU

Open starry-eyed-art opened this issue 5 years ago • 4 comments

Issue Description

MetricSearcher findOffset eating 5%~15% CPU,because of DataInputStream#readLong image-20200417170907416 image-20200417170544934

bug report

Describe what happened

当MetricSearcher工作时,会占用5%~15%的CPU。访问量小的应用没有这种问题,但是部分访问量较高的应用会偶发这个问题,具体访问量是比如一分钟20万次被Sentinel管控的请求。我们有一些用应用单机QPS超过4万,这个问题便会持续出现

When MetricSearcher works, it consumes 5% to 15% of the CPU. There is no such problem with applications with small traffic, but some applications with high traffic will have this problem occasionally. The specific traffic is for example 200,000 requests per minute controlled by Sentinel. We have some applications with a single machine QPS exceeding 40,000, this problem will continue to appear

Describe what you expected to happen

我们已经在生产环境大规模使用了Sentinel,并且已经基于Sentinel做了内部的版本。官方是否有可能优化这个问题,确实单个线程占用这么高的CPU很难接受。

We have used Sentinel on a large scale in the production environment, and have already made an internal version based on Sentinel. Is it possible for the official to optimize this problem, it is indeed difficult to accept that a single thread occupies such a high CPU.

How to reproduce it (as minimally and precisely as possible)

分析工具:top+jstack+IBMThread And Monitor Dump Analyzer

Tell us your environment

Sentinel V1.6.3

starry-eyed-art avatar Apr 17 '20 09:04 starry-eyed-art

这个官方有优化计划吗

hezhaoye avatar Jun 16 '22 03:06 hezhaoye

欢迎社区一起分析优化,结合一些 profiler 信息

sczyh30 avatar Jun 17 '22 01:06 sczyh30

这个官方有优化计划吗

可以考虑将监控数据通过别的方式进行收集,比如说将监控数据写入kafka之类的地方,避免监控数据落盘带来的IO开销

justlau avatar Mar 29 '23 12:03 justlau