petanne

Yes, we use physical machines, not cloud. Sorry, there are not enough machines to replace 48 cores with 3 * 16 cores. > Could you please share an example of...

Thank you for your careful answer. > Why would you do that? Disabling caches also increases the amount of work data blocks scanning. Normally, some previously read blocks are cached...

From this issue until now, we use `vmctl` to import Prometheus historical data, and limit the import rate. There is only a short time in between when no import is...

> So if you're using some kind of benchmark for generating the ingested data - let us know, so we can try to reproduce it. Sorry, no benchmark data.The time...

Hi, @valyala , Could you help analyze it?

> @petanne , could you try building `vmstorage` from the commit [fbeebe4](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/fbeebe4869892a9dd3c684da5fbaa0a6c511a651) and verify whether this helps resolving the issue? I build [fbeebe4](https://github.com/VictoriaMetrics/VictoriaMetrics/commit/fbeebe4869892a9dd3c684da5fbaa0a6c511a651), the problem still occurs.  I found...

@valyala , I reproduced this problem again. 1. Stop all of 2 vmalert instance. 2. Upgrade all vmstorage to https://github.com/VictoriaMetrics/VictoriaMetrics/commit/fbeebe4869892a9dd3c684da5fbaa0a6c511a651 . 3. Start one vmalert instance. Red in graphs. `-evaluationInterval=30s`....

@valyala , it works, but a new problem arises. Upgrade 13.vmstorage to `v1.85.2-cluster`, other vmstorage nodes are still https://github.com/VictoriaMetrics/VictoriaMetrics/commit/fbeebe4869892a9dd3c684da5fbaa0a6c511a651 .   Refer to the graph below for the historical...

Add to last comment, stopped the data import and the CPU problem still occurs. Then change `-evaluationInterval=10s` to `-evaluationInterval=30s` on 1.vmalert and restart it, CPU back to normal.

Me too. Are you resolve it?