Thanos Receiver: Router/Ingestor setup no longer returns `thanos_receive_write_timeseries_*` and `thanos_receive_write_samples_*` metrics with thanos v0.35.1

[roth-wine]

Thanos, Prometheus and Golang version used: quay.io/thanos/thanos:v0.35.1

Object Storage Provider:

What happened: When we updated to thanos v0.35.1, the metrics for thanos_receive_write_timeseries_* and thanos_receive_write_samples_* suddenly disappeared.

What you expected to happen: The metrics should still be available after the update to thanos v0.35.1.

How to reproduce it (as minimally and precisely as possible): To reproduce the problem, you will need to deploy a receiver setup with a dedicated router and ingestor. The router should then output the above metrics via the /metrics endpoint. However, since thanos 0.35.1 it no longer does this.

Anything else we need to know: I looked through the changelog and found the merge request 7373. If I build the v0.35.1 branch without the merge request 7373, the metrics are provided again by the /metrics endpoint.

[GiedriusS]

Disappeared where exactly? On the router? If yes then my understanding is that the router isn't writing anything itself hence those metrics shouldn't have ever existed in the first place.

[roth-wine]

@GiedriusS Correct, the metric only appears on the router. Even with other Thanos versions, it never appeared on the ingestor.

[jnyi]

+1 see this issue in our setup, left a comment in https://github.com/thanos-io/thanos/pull/7373#issuecomment-2146325916

[kevwashi]

Also seeing this issue in 0.36.1