Altinity
ClickHouse retries on not-ready servers because of `PublishNotReadyAddresses`
If a server failed to start, DNS should fail to resolve (server name does not correspond to any IP address), so as to tell the initiator ClickHouse server to avoid connecting to this server.
With PublishNotReadyAddresses=true, the initiator ClickHouse server believes a failed server should exist and makes several retries to connect it. Thus when there is any failed server, all subsequent queries will slow down for several seconds to wait for the connection retries to run out.
PublishNotReadyAddresses here https://github.com/Altinity/clickhouse-operator/blob/69d5e39807e13feef00f47434d8a9fd9f4f0430f/pkg/model/chk/creator/service.go#L227
As k8s docs says:
The primary use case for setting this field is for a StatefulSet's Headless Service to propagate SRV DNS records for its Pods for the purpose of peer discovery
But a ClickHouse cluster does not need peer discovery because it is designed as multi-master architecture, where peer addresses are resolved from their hostnames before query execution (perhaps with DNS cache). An IP address of failed server does not make it better.
In environments where administrators/users does not have the access to a incorrectly configured /etc/resolv.conf on host machine, not-ready servers may cause queries on initiator server to hang on DNS timeouts. This can be a workaround, but may not be turned on by default for the reason above.
For example, if in an isolated network where /etc/resolv.conf on host machine is configured to the Google DNS server 8.8.8.8, but ClickHouse administrators/users can not modify it. PublishNotReadyAddresses=true becomes a choice.