Receive: compaction failure causing query returning irrelevant time series

[jnyi]

Hi Team,

We've been enabled OOO for a few months, and started to see a weird behavior that when 1 or 2 pods have compaction failures, the query returns irrelevant data, appreciate any insights:

This caused false positive firing alerts in our setup with other baselines:

Thanos, Prometheus and Golang version used:

Thanos: v0.35.1 Go: go1.21.11

Object Storage Provider:

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Full logs to relevant components:

Anything else we need to know:

[yeya24]

Where did the compaction failure happen. Did you mean Compactor or Receiver?

[jnyi]

It is receiver, see more discussions here: https://cloud-native.slack.com/archives/CK5RSSC10/p1719260674748149