Julien

isnt quantile_over_time(0.5, x) this?

What is that graph representing exactly? It goes from 100gb to almost 0?

Could you share screenshots of the benchmark dashboard: https://grafana.com/grafana/dashboards/12054 ? Thanks

I am impressed that that prometheus works at all. Thats quite a scale!

You said that the logs do not matter, but I'd like to see the component=tsdb logs that you have. Typically: ``` level=info ts=2020-07-27T21:00:58.620Z caller=compact.go:495 component=tsdb msg="write block" mint=1595872800000 maxt=1595880000000 ulid=01EE916E4GPYRKQ7Y3V23DE6N7...

@codesome Like this, I'd say that in 2.19 we are taking a lock for too long during GC() ? Maybe that is linked to the new head mmap chunks?

There are no gaps in the dashboards. Is that coming from the same prometheus server? Only the statsd jobs are missing datapoints?

The metrics with gaps, do they come from recording rules?

Do they expose timestamps on metrics?

Yes, in particular what is the scrape duration of the job? How many metrics in each one of those jobs? Is there metrics relabel configs?