azurerm_kubernetes_cluster: dynamic kubelet_identity forces recreation

[amarkevich]

Is there an existing issue for this?

• [X] I have searched the existing issues

Community Note

• Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.6.6

AzureRM Provider Version

3.87.0

Affected Resource(s)/Data Source(s)

azurerm_kubernetes_cluster

Terraform Configuration Files

data "azurerm_user_assigned_identity" "umi" {
  count               = var.identity_name != null && var.identity_resource_group_name != null ? 1 : 0
  name                = var.identity_name
  resource_group_name = var.identity_resource_group_name
}

resource "azurerm_kubernetes_cluster" "this" {
  dynamic "kubelet_identity" {
    for_each = toset(data.azurerm_user_assigned_identity.umi)
    content {
      client_id                 = kubelet_identity.key.client_id
      object_id                 = kubelet_identity.key.principal_id
      user_assigned_identity_id = kubelet_identity.key.id
    }
  }
...

Debug Output/Panic Output

# module.aks.azurerm_kubernetes_cluster.this must be replaced
-/+ resource "azurerm_kubernetes_cluster" "this" {
      ~ kubelet_identity {
          ~ client_id                 = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" # forces replacement -> (known after apply) # forces replacement
          ~ object_id                 = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" # forces replacement -> (known after apply) # forces replacement
          ~ user_assigned_identity_id = "/subscriptions/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/resourceGroups/rg-NAME/providers/Microsoft.ManagedIdentity/userAssignedIdentities/IDENTITY_NAME" # forces replacement -> (known after apply) # forces replacement
        }

Expected Behaviour

provider should detect that kubelet_identity values are same

Actual Behaviour

resource azurerm_kubernetes_cluster replaced each apply

Steps to Reproduce

No response

Important Factoids

No response

References

No response

[amarkevich]

Workaround:

  dynamic "kubelet_identity" {
    for_each = {
      for umi in data.azurerm_user_assigned_identity.umi: umi.id => umi
    }
    content {
      client_id                 = kubelet_identity.value.client_id
      object_id                 = kubelet_identity.value.principal_id
      user_assigned_identity_id = kubelet_identity.value.id
    }
  }

then 'apply' works as expected but during 'plan' shows

      - kubelet_identity {
          - client_id                 = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" -> null
          - object_id                 = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX" -> null
          - user_assigned_identity_id = "/subscriptions/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/resourceGroups/rg-NAME/providers/Microsoft.ManagedIdentity/userAssignedIdentities/IDENTITY_NAME" -> null
        }

Another question - why 'identity' accept UMI id but 'kubelet_identity' needs 'client_id' & 'object_id' configured, result is the same:

  "identity": {
    "delegatedResources": null,
    "principalId": null,
    "tenantId": null,
    "type": "UserAssigned",
    "userAssignedIdentities": {
      "/subscriptions/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/resourceGroups/rg-NAME/providers/Microsoft.ManagedIdentity/userAssignedIdentities/IDENTITY_NAME": {
        "clientId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
        "principalId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
      }
    }
  },
  "identityProfile": {
    "kubeletidentity": {
      "clientId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
      "objectId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
      "resourceId": "/subscriptions/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/resourceGroups/rg-NAME/providers/Microsoft.ManagedIdentity/userAssignedIdentities/IDENTITY_NAME"
    }

[ajoskowski]

I have a similar problem.

I want to switch AKS cluster from service principal to managed identities. First what I did was switching "identity" by adding to my code:

identity {
    type = "UserAssigned"
    identity_ids = [data.azurerm_user_assigned_identity.control_plane_mi.id]
  }

Secondly I wanted to add kubelet_identity (it was not set before):

kubelet_identity {
    client_id = data.azurerm_user_assigned_identity.kubelet_mi.client_id
    object_id = data.azurerm_user_assigned_identity.kubelet_mi.principal_id
    user_assigned_identity_id = data.azurerm_user_assigned_identity.kubelet_mi.id
  }

Result is:

[2024-08-26 09:54:27]       ~ kubelet_identity {
[2024-08-26 09:54:27]           ~ client_id                 = "XXXXX-XXXX-XXXX-XXXX" -> "YYYY-YYYY-YYYY-YYYY" # forces replacement
[2024-08-26 09:54:27]           ~ object_id                 = "XXXXX-XXXX-XXXX-XXXX" -> "YYYY-YYYY-YYYY-YYYY" # forces replacement
[2024-08-26 09:54:27]           ~ user_assigned_identity_id = "/subscriptions/ZZZZ-ZZZZ-ZZZZ-ZZZZ/resourceGroups/MC_BBBBBBB-BBBBB/providers/Microsoft.ManagedIdentity/userAssignedIdentities/BBBB-BBBB-BBBB-agentpool" -> "/subscriptions/ZZZZ-ZZZZ-ZZZZ-ZZZZ/resourceGroups/BBBBBBB-BBBBB-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/BBBBBBB-BBBBB-kubelet" # forces replacement
[2024-08-26 09:54:27]         }

The problem is that terraform wants to recreate a cluster - it's expected regarding to docs where we have:

The kubelet_identity block supports the following:

[client_id](https://registry.terraform.io/providers/hashicorp/azurerm/4.0.1/docs/resources/kubernetes_cluster#client_id) - (Optional) The Client ID of the user-defined Managed Identity to be assigned to the Kubelets. If not specified a Managed Identity is created automatically. Changing this forces a new resource to be created.

[object_id](https://registry.terraform.io/providers/hashicorp/azurerm/4.0.1/docs/resources/kubernetes_cluster#object_id) - (Optional) The Object ID of the user-defined Managed Identity assigned to the Kubelets.If not specified a Managed Identity is created automatically. Changing this forces a new resource to be created.

[user_assigned_identity_id](https://registry.terraform.io/providers/hashicorp/azurerm/4.0.1/docs/resources/kubernetes_cluster#user_assigned_identity_id) - (Optional) The ID of the User Assigned Identity assigned to the Kubelets. If not specified a Managed Identity is created automatically. Changing this forces a new resource to be created.

In Azure docs I see: https://learn.microsoft.com/en-us/azure/aks/use-managed-identity#update-an-existing-cluster-to-use-the-kubelet-identity This procedure works fine - it recrates only nodepools.

Why terraform behavior is totally different?