sloth
sloth copied to clipboard
Option to generate sloth yaml - using 5m record rule chaining
We have some metrics with very high cardinality, which causing recording rules for longer periods (2h+ to timeout or simply run our Prometheus instance very high on CPU. I would like to chain recording rules all based on 5m rule, without overwriting every record rule manually in the yaml.
example for 30m record rule to be based on 5 min rule (al lthe others 1h, 2h,6h,1d,3d,1w would follow the same pattern).
- record: slo:sli_error:ratio_rate30m
expr: |
sum_over_time(slo:sli_error:ratio_rate5m{sloth_id="frontend-requests-latency", sloth_service="frontend", sloth_slo="requests-latency"}[30m])
/ ignoring (sloth_window)
count_over_time(slo:sli_error:ratio_rate5m{sloth_id="frontend-requests-latency", sloth_service="frontend", sloth_slo="requests-latency"}[30m])