Support annotating pods on change for thanos receiver router and ingestor setup

[rtnpro]

Currently, thanos-receive-controller uses the same label to watch receiver pods and update them on hashring changes.

However, in a Thanos receiver router and ingestor setup (https://thanos.io/tip/proposals-accepted/202012-receive-split.md/), while we need to watch the ingestor pods to update the hashring configmap, the hashring configmap gets used in the router pods. In this scenario, the router pods need to be annotated on hashring change instead of the ingestor pods.

For thanos-receive-controller to support this Thanos receiver router+receiver setup, it needs to support an option to specify a separate pod label (different from the pod label used to watch the receiver ingestor pods) to use for annotating pods on hashring change.

[rtnpro]

Here's a draft PR: https://github.com/observatorium/thanos-receive-controller/pull/126

[rtnpro]

@matej-g I need your thoughts on this issue.

We recently implemented the proposal to setup Thanos Receiver as Router and Ingestor for making Thanos Receiver highly available.

Currently, we are using reloader controller to rotate the receiver router pods on hashring change. It works, but has it's delay.

We have tested the changes in #126 and it works faster and updates the hashring config in the router pods without rotating them.

It will be great if this changes could be implemented in upstream.

Thanks.

[miguel-cardoso-mindera]

I'm also interested in this, has there been any news?

@rtnpro appreciate your good work!

[rtnpro]

I am still waiting on a response and review from the upstream.