[Feature] Initialization Taints

[allyford]

For node initialization scenarios, customers can use taints passed to kubelet on startup. These taints will not be reconciled by AKS at runtime. Link to documentation

[htpawel]

Any update on this? @allyford

[hterik]

related: https://github.com/Azure/AKS/issues/3276 https://github.com/Azure/AKS/issues/2934

[allyford]

This feature is now released in preview. With this preview release, initialization taints has a limitation in which the taints can only be applied during cluster create and update. We are working now to remove this limitation and will be resolved before GA.

[hterik]

Thanks for the good news @allyford. :+1:
How does this feature interact with the cluster-autoscaler? The documentation says it is useful for doing node-initialization.

Node initialization taints are placed on the node at boot time and are meant to be used temporarily, such as in scenarios where you might need extra time to set up your nodes.

I believe for that to work, the startup-taints option in the cluster-autoscaler also need to be configurable, see https://github.com/Azure/AKS/issues/3276. Otherwise the autoscaler might keep producing new nodes, when it sees that pods don't get scheduled, if the first new nodes take long time to set up and clear their init-taints.

Completing that ticket and some documentation on how to achieve such a scenario together with the autoscaler would be very useful.

[UtheMan]

@hterik - init taints added to the cluster spec are automatically added to the list of taints ignored by the AKS AutoScaler. We can add more details to docs to make it clearer.

[htpawel]

@UtheMan yes, please :) Is there any deadline for init taints btw?

[UtheMan]

@htpawel init taints are now available in preview here. We are working on removing some of the limitations that the preview version has and once that is out of the way we will begin the GA process for this feature.

[luistilingue]

It is an exciting preview feature, but it doesn't work well due to this issue related to system pool:

(SystemPoolHasRestrictedTaint) Placing custom taints on system pool is not supported(except 'CriticalAddonsOnly' taint or taint effect is 'PreferNoSchedule'). Please refer to https://aka.ms/aks/system-taints for detail
Code: SystemPoolHasRestrictedTaint
Message: Placing custom taints on system pool is not supported(except 'CriticalAddonsOnly' taint or taint effect is 'PreferNoSchedule'). Please refer to https://aka.ms/aks/system-taints for detail
Target: agentPoolProfile.nodeTaints

@allyford could you check it please?

[allyford]

@luistilingue - Placing custom taints on the system node pool are still not supported. This feature does not enable system pool taints.

[luistilingue]

@allyford yes, but using the az cli I can't pass which nodepool I'm going to enable initialization taints, it enable in all cluster node pools.

Eg.

az aks update --resource-group acme --name xmtp --nodepool-initialization-taints "node.cilium.io/agent-not-ready=true:NoExecute"

[zhangchl007]

@luistilingue @allyford Azure CLI is not ready for this preview feature yet ? Please refer to the information as below. but I can update the nodeInitialization Taints via the cluster operation api

az aks update --resource-group $RESOURCE_GROUP_NAME --name $CLUSTER_NAME --node-init-taints "sku=paas:NoSchedule"

Argument '--nodepool-initialization-taints' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus The behavior of this command has been altered by the following extension: aks-preview (SystemPoolHasRestrictedTaint) Placing custom taints on system pool is not supported(except 'CriticalAddonsOnly' taint or taint effect is 'PreferNoSchedule'). Please refer to https://aka.ms/aks/system-taints for detail Code: SystemPoolHasRestrictedTaint Message: Placing custom taints on system pool is not supported(except 'CriticalAddonsOnly' taint or taint effect is 'PreferNoSchedule'). Please refer to https://aka.ms/aks/system-taints for detail Target: agentPoolProfile.nodeTaints

"nodeInitializationTaints": [ "initTaintExample=value:NoSchedule" ]

[zhangchl007]

@hterik I passed the agent profile as below when I specify the node pool, Hope AZ-CLI could be fixed soon. "agent_pool_profiles": [ { "name": "nodepool1", "count": 2, "vm_size": "Standard_DS2_v2", "max_pods": 250, "os_type": "Linux", "type": "VirtualMachineScaleSets", "enable_auto_scaling": False, "mode": "System", "vnetSubnetID": "xxxxxx", "podSubnetID": "xxxxxx" }, { "name": "testnodpool", "count": 1, "max_pods": 250, "os_type": "Linux", "osSKU": "AzureLinux", "type": "VirtualMachineScaleSets", "enable_auto_scaling": False, "vm_size": "Standard_D4as_v5", "mode": "User", "vnetSubnetID": "xxxxxx", "podSubnetID": "xxxxxx", "nodeInitializationTaints": [ "sku=paas:NoSchedule" ] }

$ az aks nodepool show --resource-group $RESOURCE_GROUP_NAME --cluster-name $CLUSTER_NAME --name $NODE_POOL_NAME --query "nodeInitializationTaints" The behavior of this command has been altered by the following extension: aks-preview [ "sku=paas:NoSchedule" ]

NODE_POOL_NAME=spotazlinux1 :$ az aks nodepool show --resource-group $RESOURCE_GROUP_NAME --cluster-name $CLUSTER_NAME --name $NODE_POOL_NAME --query "nodeInitializationTaints" The behavior of this command has been altered by the following extension: aks-preview

[allyford]

GA not currently planned for Initialization taints. Next update to come in Sept 2025.

[zirkome]

Hi @allyford, We haven't heard from you this September, is there any update you can give for GA? Thank you!

[robbo10]

Is there any update on this?

The documentation is very misleading per - https://learn.microsoft.com/en-us/azure/aks/use-node-taints

The node initialization taints you specify apply to all of the node pools in the cluster. To apply the initialization taint to a specific node, you can use an ARM template instead of the CLI.

Yet throughout the comments above you are saying the system node pool only accepts the critical addons taint??

This is a very over complicated process for people looking to simply add/remove a taint on node start up.