thanos
thanos copied to clipboard
thanos-io
Thanos Ruler: The record and the result of its expression calculated in Thanos Query show significant differences.
Thanos, Prometheus and Golang version used:
| Version | 0.32.5 |
|---|---|
| Revision | 750e8a94eed5226cd4562117295d540a968c163c |
| Branch | HEAD |
| BuildUser | root@053ebc7b5322 |
| BuildDate | 20231019-04:13:41 |
| GoVersion | go1.21.3 |
Object Storage Provider: no Object Storage， used SSD
What happened:
The record calculated by Thanos Ruler and the result of its expression calculated in Thanos Query show significant differences.
How to reproduce it (as minimally and precisely as possible): not all records have significant differences. Full logs to relevant components: thanos-ruler no error logs
Anything else we need to know:
The accuracy of the record rule with regards to original data will depend on how often you evaluate the rule and will be impacted also by how often the data is sampled. Can you provide this information?
Keep in mind that these two queries are fundamentally different. They will almost never match 100%. For the raw query you have: how much that metric changed per minute. For the record, you have: a periodic evaluation of unknown interval (you didn't tell us how often you evaluate the rule) of how much that metric changed in the last minute.
You are querying slightly different series here. On the record query you have 48 series in the result, while in the raw data query you have only 4 series.
Also the time in both charts isn't aligned, so it becomes more difficult to look at discrepancies and think about why they happen.
@douglascamata The Thanos Ruler's eval-interval is set to 30s. Prometheus has a scrape_interval of 30s and is configured with remote_write to send data to Thanos Receive. Thanos Ruler queries data from Thanos Query, whose data store is Thanos Receive. I tried setting the eval-interval to 1 minute, but there was no apparent change. However, when I changed the time range for the rate function from [1m] to [2m], the fluctuations were noticeably reduced.
The Thanos Ruler's eval-interval is set to 30s. Prometheus has a scrape_interval of 30s
This, paired with the fact that you are calculating the rate every 1m is not a good idea. A lot can happen because the periodicity of the Ruler and the Prometheus are not aligned (they don't happen at the same time) and your rate has a small enough interval that it might not have 2 points to calculate an actual rate.
My recommendation is to bump the eval interval and the rate interval to at least double or triple the scrape interval.
This is not a problem in Thanos.
Thank you for your help! I set the evaluation interval of the rule to 90 seconds and the rate interval to 2 minutes; it seems that increasing the rate interval has the most noticeable effect.
