prometheus icon indicating copy to clipboard operation
prometheus copied to clipboard

Refactor rules seriesInPreviousEval

Open dishonono opened this issue 2 years ago • 1 comments

This PR refactors the handling of stale marks in rules.

Currently, time-series of every rule’s iteration are stored in a seriesInPreviousEval, an array of map[string]labels.Labels, which is a direct member of the rule. The map' keys are string made from the labels’ bytes - a data duplication. These labelsets of the series are stored not only during the rule group evaluation, but also between iterations (this is needed so they can later be compared with the current iteration, and any stale series can be marked as such). This makes seriesInPreviousEval a large consumer of heap space.

This PR refactors all the handling of the stale marks to a separate interface and implementation. The default implementation uses map[string]bool (Affectively a hash set). The labels can be reconstructed from the string when needed (for that FromBytes method was added to labels type).

A nonstandard implementation of that interface can be configured in ManagerOptions. This opens the options to offload the time-series to disk, which is possibly only relevant to systems running high load of rules (such as cortex ruler).

dishonono avatar Oct 26 '23 12:10 dishonono

Hi, and apologies that your PR went without comment for so long. I must say I've never seen seriesInPreviousEval show up in Prometheus memory profiles; could you open an issue giving details of the circumstances under which you see this?

(Also there is now a third implementation of Labels which would complicate your code a little more)

bboreham avatar Aug 27 '24 11:08 bboreham

Hello from the bug-scrub!

Apologies again that this idea hasn't really received any discussion, but without serious justification we could not consider such a change. If you do come back to it we can easily re-open. Thanks again.

bboreham avatar Feb 18 '25 12:02 bboreham