Pod Disruption Budget only allows 1 pod offline across entire installation

[widders]

We have a setup that results in us having multiple clickhouse pods on one kubernetes host from different clusters but the pod disruption budget being generated only allows one pod to be offline which means we can't evict the pods when doing maintenance on those nodes. Simplified below but from a service perspective any of the nodes should be safe to take down but with the disruption budget it is not possible to evict all clickhouse pods from any node.

| Node1  | Node2  | Node3  |
| main-0 | main-1 |        |
|        | int-0  | int-1  |
| ext-0  | ext-1  | ext-2  |

I can't seem to find anyway to configure how the budgets are generated, am I missing something? We'd like to have them per cluster (I guess per shard would be more appropriate but we're not using sharding at the moment)

[alex-zaitsev]

Hi @widders , PodDisruptionBudget has the following selector:

spec:
  selector:
    matchLabels:
      clickhouse.altinity.com/app: chop
      clickhouse.altinity.com/chi: <chi_name>
      clickhouse.altinity.com/namespace: <chi_namespace>
  maxUnavailable: 1

It should work across different CHI instances. Do you mean you have several clusters inside one CHI, and this is what causes issues?

[widders]

Yes exactly, we deploy three clusters from one ClickhouseInstallation so those selectors match across all three clusters.

FYI The main cluster feeds to internal and external with replicated tables.

[godwhoa]

+1 Would like to see pod disruption budget on per-shard bases.

[alex-zaitsev]

Fixed in 0.20.1

[alex-zaitsev]

https://github.com/Altinity/clickhouse-operator/releases/tag/release-0.20.1