Giedrius Statkevičius

I was able to reproduce this on 0.22.0. Interesting bug. Thanks for the report! Help welcome.

I suppose a good start would be to add failing tests. A suitable location seems to be the e2e tests: https://github.com/thanos-io/thanos/tree/main/test/e2e.

I have played around with the new memcached support and my anecdotal evidence says that this happens when memcached cannot keep up because not enough memory has been allocated to...

Thanks for trying it out and the repro! Should be easy to reproduce in e2e tests. Help wanted.

Does it help if you increase the step when there are gaps? Do you run query-frontend to help with the alignment of the step? Perhaps Thanos Store starts giving you...

I'm not sure that I am able to reproduce this. Also, I have checked out the source code and looks like the whole query get used as part of the...

Just noticed this too, reopening. * `avg (rate(node_cpu_seconds_total{mode="user",hostname=~".*foo.*"}[1m])) * 100` -> cached * `avg without (cpu) (rate(node_cpu_seconds_total{mode="user",hostname=~".*foo.*"}[1m])) * 100` -> not cached

Actually, I take those words back. Cannot reproduce this now. Probably the items from Memcached get evicted when I sent such heavy queries originally and/or there were intermittent cache connectivity...

Still relevant AFAIK.

I think for this we'd need query-frontend + Query + Store + Sidecar + Prometheus with a reverse proxy in front of query-frontend. Feel free to work on Ansible playbooks,...