cortex-helm-chart icon indicating copy to clipboard operation
cortex-helm-chart copied to clipboard

memberlist svc does not publish endpoint when not ready

Open nschad opened this issue 3 years ago • 0 comments

recently I deployed just the compactor with the "shuffle-sharding" tec from https://github.com/cortexproject/cortex/pull/4433.

The compactor would not come up due to the memberlist not publishing non-ready endpoints

ts=2022-02-22T10:48:56.126638511Z caller=memberlist_logger.go:74 level=warn msg="Failed to resolve cortex-fast-compact-memberlist: lookup cortex-fast-compact-memberlist on 10.64.0.10:53: no such host"

Only after I set

publishNotReadyAddresses: true

in the memberlist headless service the compactor would come up and join the ring

level=info ts=2022-02-22T10:50:23.815845998Z caller=memberlist_client.go:542 msg="joined memberlist cluster" reached_nodes=1

I'm currently not sure on the best way to solve that, maybe we should just allow the service to publish NonReadyAddresses.

nschad avatar Feb 22 '22 11:02 nschad