cluster-api-provider-azure

cluster-api-provider-azure copied to clipboard

/kind bug

What steps did you take and what happened: A cluster with MachinePools and externally managed autoscaler is necessary for this bug to appear.

The reconciliation loop for AzureMachinePool does not automatically refresh bootstrap tokens once they get rotated. Writing of new bootstrap tokens into custom data is done only when there is a surge change or the VMSS model changes. When scaling a VMSS manually or externally via cluster-autoscaler set to provider azure, the token in the VMSS custom data may already be outdated and therefore the new node can not join the cluster.

I believe there might be two separate issues:

1. patchVMSSIfNeeded does not verify if custom data changed and therefore does not update it. To do this, we'd need to store e.g. a hash of custom data in the AzureMachinePool.Status and compare the hashes.
2. auto rotation of Kubeadm bootstrap tokens does not result in a reconciliation of AzureMachinePool. Only a change in MachinePool leads to reconciliation of AzureMachinePool and might result in updating the VMSS custom data.

What did you expect to happen: When Kubeadm bootstrap token refreshes, the VMSS custom data gets updated automatically and new nodes can join without issues.

Anything else you would like to add: I'm aware that the prerequisites (MachinePools and externally managed cluster autoscaler) are a special case. I'm not sure how many users want to have an externally managed autoscaler and I'm personally testing whether I can switch to the cluster-api cluster-autoscaler provider. However, I do think custom data changes should be considered in patchVMSSIfNeeded.

To solve point two, I believe the AzureMachinePool controller would need to watch KubeadmConfig and then kick off a reconciliation. This might be even worth to consider in CAPI itself, since I expect this to be an issue for all CAP* providers.