Ganesh Vernekar
Ganesh Vernekar
This approach is interesting and something we can consider. I would like us to wait until https://github.com/prometheus/prometheus/pull/11075 is merged because doing it the other way round will be more challenging.
@richardjennings does your scrape target specify timestamps for the metric? If not, several minutes of scrape should not create this issue. Is your scrape scraping samples of nearly 1 hour...
How long is your scrape interval? Default lookback interval is 5 mins, which should work fine with the recommended scrape interval (
I have also seen that the memory spikes during WAL replay to almost double the consumption before shutdown (no remote write). Ideally WAL replay should not take so much memory,...
We would need some profiling for WAL replay to find the pieces that is holding onto memory. I think this is a common occurrance. While it is in my interest...
I did a quick profiling of WAL replay yesterday and there was nothing that stood out as an obvious thing to fix/optimise. I am expecting https://github.com/prometheus/prometheus/pull/7229 to help in this.
> I don't think it'll make a difference Building a chunk from samples takes more allocations and temporary memory than simply taking encoded chunk bytes and sticking into the memory....
Hmmm, lint is failing for unrelated change as well. Looks like a simple fix of getting rid of a type. Would you be able to do it in this PR?...
@bwplotka it was this commit https://github.com/prometheus/prometheus/pull/11075/commits/faa90015d843f58d83e4518ad91d6d026840b662. It was mostly amplified with the distributed querying nature of Mimir and might not affect Prometheus. We have a cache for matcher->postings in mimir-prometheus...
Last few commits are also bug fixes. https://github.com/prometheus/prometheus/pull/11075/commits/710770e8a7dbad19903dd6b59fb8c53a435d70ff fixes a panic. https://github.com/prometheus/prometheus/pull/11075/commits/8daef745a03da4024523d26e8e28109e3df29f6b fixes a case where the disk was growing till it became full. (these were not tricky to identify)