cortex-helm-chart
cortex-helm-chart copied to clipboard
memberlist svc does not publish endpoint when not ready
recently I deployed just the compactor with the "shuffle-sharding" tec from https://github.com/cortexproject/cortex/pull/4433.
The compactor would not come up due to the memberlist not publishing non-ready endpoints
ts=2022-02-22T10:48:56.126638511Z caller=memberlist_logger.go:74 level=warn msg="Failed to resolve cortex-fast-compact-memberlist: lookup cortex-fast-compact-memberlist on 10.64.0.10:53: no such host"
Only after I set
publishNotReadyAddresses: true
in the memberlist headless service the compactor would come up and join the ring
level=info ts=2022-02-22T10:50:23.815845998Z caller=memberlist_client.go:542 msg="joined memberlist cluster" reached_nodes=1
I'm currently not sure on the best way to solve that, maybe we should just allow the service to publish NonReadyAddresses.