prometheus icon indicating copy to clipboard operation
prometheus copied to clipboard

Rule Manager: Only query once per alert rule when restoring alert state

Open gotjosh opened this issue 2 months ago • 0 comments

Prometheus restores alert state between restarts and updates. For each rule, it looks at the alerts that are meant to be active and then queries the ALERTS_FOR_STATE series for each alert within the rules.

If the alert rule has 120 instances (or series) it'll execute the same query with slightly different labels.

This PR changes the approach so that we only query once per alert rule and then match the corresponding alert that we're about to restore against the series-set. While the approach might use a bit more memory at start-up (if even?) the restore proccess is only ran once per restart so I'd consider this a big win.

This builds on top of https://github.com/prometheus/prometheus/pull/13974

gotjosh avatar Apr 23 '24 18:04 gotjosh