David Grant

> we don't preserve all block labels in the bucket index What labels are absent from the bucket index? Is it labels from partial blocks?

Around the time when this issue was created, Mimir also gained a log sampling facility (https://github.com/grafana/mimir/pull/5584). A bunch of errors are now being downsampled: https://github.com/grafana/mimir/blob/c0038b146c3991b1c5eb1250ece41883bed1086c/pkg/ingester/errors.go#L543-L553 (Although at the moment it...

I have surveyed the scene. With the implementation of #5584 and #5764, I have found no logs that are still egregiously abusing our systems. Both of those changes are protecting...

Let me know if I understand the full picture before I start documenting. (cc @pracucci) > The compactor is stateless but it does quite a lot of disk I/O. We...

> > The reason we use StatefulSets with compactor is that StatefulSets are configured with a persistent volume claim template and each replica gets its own persistent volume. If we...

> I wonder if we store the config first for serving the URL and then notify the modules that use it means that there's a short window to be able...

{waking up on this Monday morning 😸.} But the race with the second of sleep you highlighted would also cause it.

I guess your second of sleep just takes longer than the tests [care to wait](https://github.com/grafana/mimir/blob/91dd4380e5c82b883396dee4bb3f485ca924def6/integration/distributor_test.go#L343) for the new config to be available. 1 second, while an eternity, could also be...

Actually, I don't think that was the problem. Here's the run that failed: https://github.com/grafana/mimir/actions/runs/8974460472/job/24647018591 [test.Poll will t.Fatal](https://github.com/grafana/mimir/blob/727b9584a7a837f7fbec2bfc619ebfc9f7f404da/vendor/github.com/grafana/dskit/test/poll.go#L23-L25) if the condition isn't met after the 1 second, and that didn't happen...

The way Marshal first gets the size of the required buffer by calling sovFrontend, then later `MarshalToSizedBuffer` recomputes those value sizes again to figure out where in the byte slice...