Aliaksandr Valialkin
Aliaksandr Valialkin
@petanne , could you share graphs for the following queries on the same time range as graphs above if these graphs expose some anomalies around the CPU usage spikes time...
@petanne , thanks for the provided graphs! Could you try building and running `vmstorage` from the commit 8c08d625eeb4c5b224b031f27693b62bc4403602 ?
@petanne , the commit 8c08d62 has been included in [v1.85.2](https://github.com/VictoriaMetrics/VictoriaMetrics/releases), so you can try the official `vmstorage` from the `v1.85.2-cluster` tag instead of building it on your own. It also...
@petanne , thanks for providing useful graphs! It looks like the CPU usage highly correlates the the number of parts of the `storage/inmemory` type. Big number of data parts may...
Initial version of Kuma service discovery aka [kuma_sd_config](https://docs.victoriametrics.com/sd_configs.html#kuma_sd_configs) is available starting from the commit 9fbd45a22fdba49fd496aa6d2760b1cd4e44a46f . Thanks to @Amper for the implementation! It will be included in the next release....
`vmagent` supports [Kuma service discovery](https://docs.victoriametrics.com/sd_configs.html#kuma_sd_configs) starting from [v1.88.0](https://docs.victoriametrics.com/CHANGELOG.html). Closing the feature request as done.
Scrapers are stopped in parallel in the latest release of vmagent (v1.83.1) according to the following code: https://github.com/VictoriaMetrics/VictoriaMetrics/blob/e014467f4255e5ab31752334415a1176149da931/lib/promscrape/scraper.go#L378-L392 This means that the total wait time for removing obsolete scrapers should...
@chadr123 , thanks for the investigation the root cause of the issue! Indeed, `vmagent` waits until currently running scrapers are stopped before starting new scrapers. But it doesn't interrupt in-flight...
Side note for @zekker6 : the https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3468 is somewhat related to this one - in this issue the old Consul service discovery workers are blocked for up to 10 minutes...
@chadr123 , @zekker6 is working on this issue. Possible related issue - https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3747