Julien

can we have the output of the instant query http://127.0.0.1:9090 up{job=~"statsd|prometheus.*"}[20m] around the time of the issue? e.g. at 9.10.

I am looking in the "console" view, for the exact timestamps. You can paste that into a gist maybe? Something like: 

Thank you. It seems like indeed your prometheus is overloaded when we truncate the head. I will investigate later, but there are 1971 scrapes wich seems to lake > 30s....

So far it seems related to the trucate of the head, possibly of the new mmaped chunks. Could you provide us a goroutine dumps when there is no metrics? There's...

As I said a goroutine dump during one of those timeframes might help us.

cc @bwplotka

We have looked at this pull request during our bug scrub. By now, we aren't even sure if it is worth changing the names. People are really used to the...

That would be some work but could we have somehow relabelling to "duplicate" metrics/configs ? usecases: when moving from wmi_ to windows_ metrics, you would have both the old and...

We can check the disk size every time the BeyondSizeRetention function is called when a percentage is given. This way, we can ensure that we always have an up-to-date value...

I am wondering it it is worth complicating this code now, since there is a consensus to make remote_write transactional, so all this code would change anyway. https://docs.google.com/document/d/1vhXKpCNY0k2cbm0g10uM2msXoMoH8CTwrg_dyqsFUKo/edit