chi Clickhouse pods get scheduled on Zookeeper nodes

[Zman94]

In the documentation, Zookeeper pods require "separate servers from those used for ClickHouse. Zookeeper has poor performance when installed on the same node as ClickHouse." altinity documentation.

When installing the operator on a cluster, I see zookeeper pods are, by default, deployed on clickhouse-zookeeper nodes

      spec:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: service
                  operator: In
                  values:
                  - clickhouse-zookeeper
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - zookeeper
              topologyKey: kubernetes.io/hostname

I don't see any such affinity rules on clickhouse pods in the chi-clickhouse, resulting in pods scheduled on the same node as zookeeper pods:

chi-clickhouse-clickhouse-3-1-0    1/1     Running   0          278d   10.199.59.27    aks-zookeeper-63222871-vmss000000    <none>           <none>
chi-clickhouse-clickhouse-4-0-0    1/1     Running   0          278d   10.199.58.192   aks-clickhouse-10665974-vmss000003   <none>           <none>
chi-clickhouse-clickhouse-4-1-0    1/1     Running   0          278d   10.199.56.77    aks-clickhouse-10665974-vmss00000y   <none>           <none>
chi-clickhouse-clickhouse-5-0-0    1/1     Running   0          278d   10.199.56.38    aks-general-95678037-vmss00000w      <none>           <none>
chi-clickhouse-clickhouse-5-1-0    1/1     Running   0          278d   10.199.56.237   aks-zookeeper-63222871-vmss000004    <none>           <none>

Additionally, I see no configuration settings to support assigning chi-clickhouse pods in only clickhouse nodes. I see only zone settings and OnePerHost settings. altinity docs

Is this something that should be set by default? If the documentation requires clickhouse pods to be scheduled on different nodes than zookeeper pods, I would like to see a default anti-affinity rule on chi-clickhouse pods to not schedule on zookeeper nodes

[alex-zaitsev]

@Zman94 , operator supports full Kubernetes specification for affinity/anti-affinity rules, tolerations and nodeSelectors. So depending on your Kubernetes setup you may use the right toolset. We usually use a combination of the following:

• taint nodes for specific workloads and add tolerations to CHI
• use nodeSelector to schedule on specific instance type in public clouds
• use affinity rules or zone selectors to schedule in specific availability zones. Same can be applied for ZooKeeper as well.