terraform-provider-azurerm
terraform-provider-azurerm copied to clipboard
hashicorp
User Assigned Identity Removal Ineffective in VMSS Identity Block
Is there an existing issue for this?
- [X] I have searched the existing issues
Community Note
- Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.
Terraform Version
1.2.1
AzureRM Provider Version
3.93.0
Affected Resource(s)/Data Source(s)
azurerm_linux_virtual_machine_scale_set
Terraform Configuration Files
# Reference: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/linux_virtual_machine_scale_set
terraform {
required_version = ">=0.12"
}
locals {
first_public_key = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+wWK73dCr+jgQOAxNsHAnNNNMEMWOHYEccp6wJm2gotpr9katuF/ZAdou5AaW1C61slRkHRkpRRX9FA9CYBiitZgvCCz+3nWNN7l/Up54Zps/pHWGZLHNJZRYyAB6j5yVLMVHIHriY49d/GZTZVNB8GoJv9Gakwc/fuEZYYl4YDFiGMBP///TzlI4jhiJzjKnEvqPFki5p2ZRJqcbCiF4pJrxUQR/RXqVFQdbRLZgYfJ8xGB878RENq3yQ39d8dVOkq4edbkzwcUmwwwkYVPIoDGsYLaRHnG+To7FvMeyO7xDVQkMKzopTQV8AuKpyvpqu0a9pWOMaiCyDytO7GGN [email protected]"
}
provider "azurerm" {
features {}
}
resource "azurerm_resource_group" "example" {
name = "example-resources"
location = "West US"
}
resource "azurerm_virtual_network" "example" {
name = "example-network"
resource_group_name = azurerm_resource_group.example.name
location = azurerm_resource_group.example.location
address_space = ["10.0.0.0/16"]
}
resource "azurerm_subnet" "internal" {
name = "internal"
resource_group_name = azurerm_resource_group.example.name
virtual_network_name = azurerm_virtual_network.example.name
address_prefixes = ["10.0.2.0/24"]
}
resource "azurerm_user_assigned_identity" "vmss_identity_1" {
resource_group_name = azurerm_resource_group.example.name
location = azurerm_resource_group.example.location
name = "test-vmss-identity-1"
}
resource "azurerm_user_assigned_identity" "vmss_identity_2" {
resource_group_name = azurerm_resource_group.example.name
location = azurerm_resource_group.example.location
name = "test-vmss-identity-2"
}
resource "azurerm_linux_virtual_machine_scale_set" "example" {
name = "example-vmss"
resource_group_name = azurerm_resource_group.example.name
location = azurerm_resource_group.example.location
sku = "Standard_F2"
instances = 1
admin_username = "adminuser"
identity {
type = "UserAssigned"
# I first created the identity_ids with 2 identities and it created the both and asssociated with the VMSS, then I ran it with just 1 to see if it will unassign 1 and it did not.
identity_ids = [azurerm_user_assigned_identity.vmss_identity_1.id , azurerm_user_assigned_identity.vmss_identity_2.id]
# identity_ids = [azurerm_user_assigned_identity.vmss_identity_1.id]
}
admin_ssh_key {
username = "adminuser"
public_key = local.first_public_key
}
source_image_reference {
publisher = "Canonical"
offer = "0001-com-ubuntu-server-jammy"
sku = "22_04-lts"
version = "latest"
}
os_disk {
storage_account_type = "Standard_LRS"
caching = "ReadWrite"
}
network_interface {
name = "example"
primary = true
ip_configuration {
name = "internal"
primary = true
subnet_id = azurerm_subnet.internal.id
}
}
}
Debug Output/Panic Output
https://gist.github.com/saketh-cloudknox/7b78170917f119c259cc16cc7a6428f6
Expected Behaviour
The user assigned identity that was removed from the identity block should have been removed from the VMSS.
Actual Behaviour
The user assigned identity that was removed from the identity block remained attached to the VMSS.
Steps to Reproduce
User Assigned Identity Removal Ineffective in VMSS Identity Block
Reproduction Steps
- Run
terraform apply --auto-approve. Terraform shows that 2 User assigned identities will be assigned to the VMSS. This is reflected in the Azure Portal. - Remove one of the
identity_idsfrom theidentityblock under theazurerm_linux_virtual_machine_scale_set. - Run
terraform apply --auto-approve. Terraform shows that 1 User assigned identity will be unassigned from the VMSS as per the plan. - Removal is not reflected in the azure portal or via CLI. VMSS still has BOTH user assigned identities associated with it.
Important Factoids
Azure Production
References
No response
Azure VMSS Identity Update Terraform Issues
Hi all, I did some digging today into why this is happening and I think this should summarize the issue and the potential fix. Looking to get some opinions on it!
1. Problem Statement
The azurerm Terraform Provider does not properly remove identities from the VMSS when removing them from the Terraform configuration. The Terraform Provider tries to patch the VMSS using only the identities that are present in the Terraform configuration. When existing identities are patched with the same values, the API does not remove any identities not present in the PATCH request. The identities are passed in as a map where the key is the identity and identities to be removed have their value set to null, while identities to be kept/added are set to {}
1.1. REST API Behavior
This request will remove the userAssignedIdentity with the id /subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test-vmss-identity-2. It will not remove any other identities that are present in the VMSS.
PATCH https://management.azure.com/subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.Compute/virtualMachineScaleSets/example-vmss?api-version=2023-09-01
{
"identity": {
"type": "userassigned",
"userAssignedIdentities": {
"/subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test-vmss-identity-2": null
}
}
}
This request will not remove any identities from the VMSS, even though the identity with the id /subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test-vmss-identity-2 is not present in the PATCH request.
PATCH https://management.azure.com/subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.Compute/virtualMachineScaleSets/example-vmss?api-version=2023-09-01
{
"identity": {
"type": "userassigned",
"userAssignedIdentities": {
"/subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test-vmss-identity-1": {}
}
}
}
1.2 Terraform Provider Behavior
If the initial state of the VMSS looks like the following in terraform, then terraform will ensure that the VMSS has both idenitites attached to it.
resource "azurerm_linux_virtual_machine_scale_set" "example" {
name = "example-vmss"
resource_group_name = azurerm_resource_group.example.name
location = azurerm_resource_group.example.location
sku = "Standard_F2"
instances = 1
admin_username = "adminuser"
identity {
type = "UserAssigned"
identity_ids = [azurerm_user_assigned_identity.vmss_identity_1.id , azurerm_user_assigned_identity.vmss_identity_2.id]
}
admin_ssh_key {
username = "adminuser"
public_key = local.first_public_key
}
source_image_reference {
publisher = "Canonical"
offer = "0001-com-ubuntu-server-jammy"
sku = "22_04-lts"
version = "latest"
}
os_disk {
storage_account_type = "Standard_LRS"
caching = "ReadWrite"
}
network_interface {
name = "example"
primary = true
ip_configuration {
name = "internal"
primary = true
subnet_id = azurerm_subnet.internal.id
}
}
}
The user then removes the identity with the id azurerm_user_assigned_identity.vmss_identity_2.id from the VMSS configuration and runs terraform apply.
resource "azurerm_linux_virtual_machine_scale_set" "example" {
name = "example-vmss"
resource_group_name = azurerm_resource_group.example.name
location = azurerm_resource_group.example.location
sku = "Standard_F2"
instances = 1
admin_username = "adminuser"
identity {
type = "UserAssigned"
identity_ids = [azurerm_user_assigned_identity.vmss_identity_1.id]
}
admin_ssh_key {
username = "adminuser"
public_key = local.first_public_key
}
source_image_reference {
publisher = "Canonical"
offer = "0001-com-ubuntu-server-jammy"
sku = "22_04-lts"
version = "latest"
}
os_disk {
storage_account_type = "Standard_LRS"
caching = "ReadWrite"
}
network_interface {
name = "example"
primary = true
ip_configuration {
name = "internal"
primary = true
subnet_id = azurerm_subnet.internal.id
}
}
}
The terraform provider will then patch the VMSS with the following request (according to the debug logs).
PATCH /subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.Compute/virtualMachineScaleSets/example-vmss?api-version=2023-03-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.21.6 (amd64-darwin) go-autorest/v14.2.1 tombuildsstuff/kermit/v0.20240122.1123108 compute/2023-03-01 HashiCorp Terraform/1.2.1 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/3.93.0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 446
Content-Type: application/json; charset=utf-8
X-Ms-Correlation-Request-Id: 0794a696-9d83-f3e7-07c6-d6569402a777
Accept-Encoding: gzip
{
"identity": {
"type": "UserAssigned",
"userAssignedIdentities": {
"/subscriptions/<azure-subscription-id>/resourceGroups/example-resources/providers/Microsoft.ManagedIdentity/userAssignedIdentities/test-vmss-identity-1": {}
}
},
"properties": {
"upgradePolicy": {
"mode": "Manual"
},
"virtualMachineProfile": {
"storageProfile": {
"imageReference": {
"offer": "0001-com-ubuntu-server-jammy",
"publisher": "Canonical",
"sku": "22_04-lts",
"version": "latest"
}
}
}
}
}
Based on the API behvaior noted in section 1.1, the VMSS will still have the identity with the id azurerm_user_assigned_identity.vmss_identity_2.id attached to it since it did not set the identity to null in the PATCH request and only set the identity with the id azurerm_user_assigned_identity.vmss_identity_1.id to {}.
2. Proposed Solution
2.1. Update Terraform Provider
In /internal/services/compute/linux_virtual_machine_scale_set_resource.go the Update method should be updated to remove any identities that are not present in the Terraform configuration.
The Update method associated with identities currently has this code block:
if d.HasChange("identity") {
identityRaw := d.Get("identity").([]interface{})
identity, err := expandVirtualMachineScaleSetIdentity(identityRaw)
if err != nil {
return fmt.Errorf("expanding `identity`: %+v", err)
}
update.Identity = identity
}
It is not adding the entry in the identity map for the identities that need to be removed from the VMSS by setting their value to null. The expandVirtualMachineScaleSetIdentity function should be updated to add the identities that need to be removed to the map with a value of null. These identities are available via the variable existing.Identity.UserAssignedIdentities which will give the existing identities that can be compared against the identities returned by expandVirtualMachineScaleSetIdentity
Hi @saketh-cloudknox, thank you for reporting this issue. However, it may not be fixed at once because I found in some situation the backend will reply an error for the identity with null value. We need to double confirm with Azure team about the reason.
PATCH
X-Ms-Correlation-Request-Id: e94fd18c-905c-a5e5-5c1c-566f79e19741
{
"identity": {
"type": "UserAssigned",
"userAssignedIdentities": {
"/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/acctestRG-OVMSS-240305172229758653/providers/Microsoft.ManagedIdentity/userAssignedIdentities/acctest2oxyl2": null,
"/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/acctestRG-OVMSS-240305172229758653/providers/Microsoft.ManagedIdentity/userAssignedIdentities/acctestoxyl2": {}
}
},
"properties": {
"virtualMachineProfile": {
"osProfile": {
"linuxConfiguration": {}
},
"storageProfile": {
"imageReference": {
"offer": "0001-com-ubuntu-server-jammy",
"publisher": "Canonical",
"sku": "22_04-lts",
"version": "latest"
}
}
}
}
}
RESPONSE
{
"error": {
"code": "InvalidParameter",
"message": "ResourceIdentity userAssignedIdentities should not have keys with null values.",
"target": "resourceIdentity.userAssignedIdentities"
}
}
Currently, there is a tricky way to temporarily unblock this issue: remove azurerm_user_assigned_identity.vmss_identity_2 block with azurerm_user_assigned_identity.vmss_identity_2.id of azurerm_linux_virtual_machine_scale_set.example together.
@ms-zhenhua
I found in some situation the backend will reply an error for the identity with null value.
This is the expected API behaviour FWIW, you must specify an empty payload rather than null - it's a consistent behaviour across multiple Azure Services
