kubernetes-sigs
OpenstackCluster multi-az controlplane nodes
/kind bug
What steps did you take and what happened: [A clear and concise description of what the bug is.]
Created a cluster on a 3AZ openstack provider, hoped for controlplane nodes to be evenly distributed among them but it is not the case. I have most of the time 2az used and sometime just 1, we need to use 3 of them 100% of the time.
What did you expect to happen:
Controlplane nodes evenly distributed on all AZs explicited in OpenstackCluster.spec.controlPlaneAvailabilityZones
Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]
Environment:
- Cluster API Provider OpenStack version (Or
git rev-parse HEADif manually built): v0.12.2 - Cluster-API version: v1.9.6
- OpenStack version: 3.14 (ovhcloud)
- Minikube/KIND version:
- Kubernetes version (use
kubectl version): 1.32.2 - OS (e.g. from
/etc/os-release): talos 1.9.2
@EmilienM @mdbooth what do you think about this? Is the multi-AZ functionality broken? We have the e2e test for this, but there we only have 2 AZs.
To add to the ticket: I add this issue with talos provider which provision all 3 control-plate nodes at the same time. Since switching to kubeadm I'm not having this anymore, all nodes are distributed among the 3 AZs
IFAIR CAPI's failure domains are only used by the kubeadm provider. The kubeadm control plane provider sets the failure domain explicitly when creating a machine. If you aren't using that controller then you'll need to achieve the same with the OpenStack scheduler.
@stephenfin is there an 'AZ anti-affinity' policy?
@stephenfin is there an 'AZ anti-affinity' policy?
Not that I'm aware of. There are server group anti-affinity policies but the use case for that is "I don't want these applications to run on the same host" not "I don't want these applications to run on the same racks/datacentre". The expected pattern for AZs is that if you want N servers spread across N AZs, then you need to make N separate requests with a different availability_zone request for each.
FWIW, specifically here: https://github.com/kubernetes-sigs/cluster-api/blob/31ea0af963eecf1938c9c7eae75491b0cece4e39/controlplane/kubeadm/internal/controllers/scale.go#L91-L94
Was running into the same issue with OpenStack Infra Provider and Talos Control Plane Provider. Not a real "fix", but was able to resolve this by leveraging Kyverno Cluster Policies. For each AZ desired for the control planes, a policy will intercept the machine creation, check if there are fewer machine's with that failure domain tagged than desired, and if so mutate the spec.failureDomain to the desired Availability Zone.
