Add kubelet_disk_type in AKS additional_node_pools

[NissesSenap]

This allows us to use Temporary disk types which should remove most of the risk of running ful on images, emptyDir and logs on AKS nodes. Solves #554

Upgrade azurerm to 2.97 to get this new feature.

The valid options for kubelet_disk_type are: OS and Temporary

[NissesSenap]

If you don't want to change your existing node pool all you need to do is to configure it to use: kubelet_disk_type = "OS"

[NissesSenap]

We have read lots and lots of docs and we don't think that the kubelet_disk_type is in preview and thus ready for production.

https://github.com/Azure/azure-rest-api-specs/blob/main/specification/containerservice/resource-manager/Microsoft.ContainerService/stable/2022-01-01/managedClusters.json

For more information: https://github.com/Azure/AKS/issues/930 https://docs.microsoft.com/en-us/rest/api/aks/agent-pools/create-or-update#kubeletdisktype

[NissesSenap]

When trying to apply the config I get the error saying that it's a preview feature. So I think we can skip this for now since it will only force us to set another config value.

│ Error: creating Node Pool: (Agent Pool Name "standard3" / Managed Cluster Name "aks-dev-we-aks1" / Resource Group "rg-dev-we-aks"): containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="PreviewFeatureNotRegistered" Message="Preview feature Microsoft.ContainerService/KubeletDisk not registered."
│

[NissesSenap]

As noted in the issue this feature have now gone GA. An interesting thing to know is that even though terraform say it can do a in-place change it seems like it isn't possible.

I ran a in-place change but after running it a second time I get the following information.

      ~ kubelet_disk_type      = "OS" -> "Temporary"

So my guess is that Azure changed back... But when I try to add a new node pool I get the same old error:

│ Error: creating Node Pool: (Agent Pool Name "standard3" / Managed Cluster Name "aks-sand-we-aks1" / Resource Group "rg-sand-we-aks"): containerservice.AgentPoolsClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="PreviewFeatureNotRegistered" Message="Preview feature Microsoft.ContainerService/KubeletDisk not registered."
│ 
│   with module.aks1.azurerm_kubernetes_cluster_node_pool.this["standard3"],
│   on .terraform/modules/aks1/modules/azure/aks/aks.tf line 74, in resource "azurerm_kubernetes_cluster_node_pool" "this":
│   74: resource "azurerm_kubernetes_cluster_node_pool" "this" {
│ 

On the release page it says that the features will roll out by 2022-04-15. I will wait until then and if it still dont work without the preview feature I will open a issue.

[NissesSenap]

I have misunderstood, I thought that custom node config ment kubelet disk type since it's a configuration that is done on the node... It's not.

Even though I can't find anything about kubelet disk type https://docs.microsoft.com/en-us/rest/api/aks/managed-clusters/create-or-update#kubeletdisktype being in preview still. I might miss something obvious in the terraform provider but I haven't had the time to look closer. @phillebaba would you have time to look through the azurerm code and see if there is some update that needs happening and that they are using the preview api instead of stable for this feature?