Subnet Service delegation continuously must be sorted

[bjornhofer]

Is there an existing issue for this?

• [X] I have searched the existing issues

Community Note

• Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.1.6

AzureRM Provider Version

3.33.0

Affected Resource(s)/Data Source(s)

azurerm_subnet

Terraform Configuration Files

resource "azurerm_resource_group" "example" {
  name = "delegation-example"
  location = "westeurope"
}

resource "azurerm_virtual_network" "example" {
  name                = "delegation-example"
  address_space       = ["192.168.0.0/20"]
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
}

// simulation of dynamic values - normaly coming from an (internal) API
locals {
  subnets = {
    "192.168.1.0/24" : {
      "name" : "subnet1",
      "cidr" : "192.168.1.0/24",
      "provider_specific" : {
        "delegations" : {
          "delegation1" : {
            "name" : "Microsoft.Web/serverFarms",
            "actions" : [
              "Microsoft.Network/virtualNetworks/subnets/action",
              "Microsoft.Network/virtualNetworks/subnets/join/action"
            ]
          }
        }
        "tags" : {}
    } }
    "192.168.2.0/24" : {
      "name" : "subnet2",
      "cidr" : "192.168.2.0/24",
      "provider_specific" : {
        "delegations" : {
          "delegation1" : {
            "name" : "Microsoft.Databricks/workspaces",
            "actions" : [
              "Microsoft.Network/virtualNetworks/subnets/action",
              "Microsoft.Network/virtualNetworks/subnets/unprepareNetworkPolicies/action",
              "Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action",
              "Microsoft.Network/virtualNetworks/subnets/join/action"
            ]
          }
        }
      },
      "tags" : {}
    }
  }
}

// create subnet and delegation 
resource "azurerm_subnet" "delegation_dynamic" {
  for_each             = local.subnets
  name                 = each.value.name
  resource_group_name  = azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = [each.value.cidr]
  dynamic "delegation" {
    for_each = contains(keys(each.value), "provider_specific") ? lookup(each.value.provider_specific, "delegations", {}) : {}
    content {
      name = delegation.key
      service_delegation {
        name    = delegation.value.name
        actions = delegation.value.actions
      }
    }
  }
  depends_on = [
    azurerm_virtual_network.example
  ]
}

// assign the same delegations static - also fails
resource "azurerm_subnet" "delegation_static1" {
  name                 = "subnet3"
  resource_group_name  = azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["192.168.3.0/24"]
  delegation {
    name = "delegation1"
    service_delegation {
      name    = "Microsoft.Web/serverFarms"
      actions = ["Microsoft.Network/virtualNetworks/subnets/action", "Microsoft.Network/virtualNetworks/subnets/join/action"]
    }
  }
}

resource "azurerm_subnet" "delegation_static2" {
  name                 = "subnet4"
  resource_group_name  = azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["192.168.4.0/24"]
  delegation {
    name = "delegation1"
    service_delegation {
      name    = "Microsoft.Databricks/workspaces"
      actions = ["Microsoft.Network/virtualNetworks/subnets/unprepareNetworkPolicies/action", "Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action", "Microsoft.Network/virtualNetworks/subnets/join/action"]
    }
  }
}

Debug Output/Panic Output

none

Expected Behaviour

After once applied, terraform should find no "difference" in the subsequent runs

Example:

terraform apply 

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no
changes are needed.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Actual Behaviour

When first applied, the delegations are properly assigned. If I rerun plan/apply - it will remove and add delegations, which are already in place (so they are removed and reapplied - in Databricks for example)

  ~ resource "azurerm_subnet" "delegation_dynamic" {
        id                                             = "/subscriptions/73a9c577-8635-4b8f-80b7-072178ee4f2a/resourceGroups/delegation-example/providers/Microsoft.Network/virtualNetworks/delegation-example/subnets/subnet1"
        name                                           = "subnet1"
        # (9 unchanged attributes hidden)

      ~ delegation {
            name = "delegation1"

          ~ service_delegation {
              ~ actions = [
                    "Microsoft.Network/virtualNetworks/subnets/action",
                  + "Microsoft.Network/virtualNetworks/subnets/join/action",
                ]
                name    = "Microsoft.Web/serverFarms"
            }
        }
    }

  # azurerm_subnet.delegation_dynamic["192.168.2.0/24"] will be updated in-place
  ~ resource "azurerm_subnet" "delegation_dynamic" {
        id                                             = "/subscriptions/73a9c577-8635-4b8f-80b7-072178ee4f2a/resourceGroups/delegation-example/providers/Microsoft.Network/virtualNetworks/delegation-example/subnets/subnet2"
        name                                           = "subnet2"
        # (9 unchanged attributes hidden)

      ~ delegation {
            name = "delegation1"

          ~ service_delegation {
              ~ actions = [
                  - "Microsoft.Network/virtualNetworks/subnets/join/action",
                  - "Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action",
                  + "Microsoft.Network/virtualNetworks/subnets/action",
                    "Microsoft.Network/virtualNetworks/subnets/unprepareNetworkPolicies/action",
                  + "Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action",
                  + "Microsoft.Network/virtualNetworks/subnets/join/action",
                ]
                name    = "Microsoft.Databricks/workspaces"
            }
        }
    }

  # azurerm_subnet.delegation_static1 will be updated in-place
  ~ resource "azurerm_subnet" "delegation_static1" {
        id                                             = "/subscriptions/73a9c577-8635-4b8f-80b7-072178ee4f2a/resourceGroups/delegation-example/providers/Microsoft.Network/virtualNetworks/delegation-example/subnets/subnet3"
        name                                           = "subnet3"
        # (9 unchanged attributes hidden)

      ~ delegation {
            name = "delegation1"

          ~ service_delegation {
              ~ actions = [
                    "Microsoft.Network/virtualNetworks/subnets/action",
                  + "Microsoft.Network/virtualNetworks/subnets/join/action",
                ]
                name    = "Microsoft.Web/serverFarms"
            }
        }
    }

  # azurerm_subnet.delegation_static2 will be updated in-place
  ~ resource "azurerm_subnet" "delegation_static2" {
        id                                             = "/subscriptions/73a9c577-8635-4b8f-80b7-072178ee4f2a/resourceGroups/delegation-example/providers/Microsoft.Network/virtualNetworks/delegation-example/subnets/subnet4"
        name                                           = "subnet4"
        # (9 unchanged attributes hidden)

      ~ delegation {
            name = "delegation1"

          ~ service_delegation {
              ~ actions = [
                  - "Microsoft.Network/virtualNetworks/subnets/join/action",
                  - "Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action",
                    "Microsoft.Network/virtualNetworks/subnets/unprepareNetworkPolicies/action",
                  + "Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action",
                  + "Microsoft.Network/virtualNetworks/subnets/join/action",
                ]
                name    = "Microsoft.Databricks/workspaces"
            }
        }
    }

Plan: 0 to add, 4 to change, 0 to destroy.

This effect can somehow be limited by removing delegation which are/might not be necessary (e.g. Microsoft.Network/virtualNetworks/subnets/action in Databricks) - which is okay/fair as Azure seems to change/remove those internally.

In addition to remove the not necessary delegation(s), the actions block MUST be sorted, if there are more than one entry (e.g. in Databricks) like this:

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "=3.33.0"
    }
  }
}

locals {
  subscription = "73a9c577-8635-4b8f-80b7-072178ee4f2a"
}

provider "azurerm" {
  subscription_id = local.subscription
  features {}
}

// basis
resource "azurerm_resource_group" "example" {
  name     = "delegation-example"
  location = "westeurope"
}

resource "azurerm_virtual_network" "example" {
  name                = "delegation-example"
  address_space       = ["192.168.0.0/20"]
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
}

// create subnet and delegation dynamic
locals {
  subnets = {
    "192.168.1.0/24" : {
      "name" : "subnet1",
      "cidr" : "192.168.1.0/24",
      "provider_specific" : {
        "delegations" : {
          "delegation1" : {
            "name" : "Microsoft.Web/serverFarms",
            "actions" : ["Microsoft.Network/virtualNetworks/subnets/action"]
          }
        }
        "tags" : {}
    } }
    "192.168.2.0/24" : {
      "name" : "subnet2",
      "cidr" : "192.168.2.0/24",
      "provider_specific" : {
        "delegations" : {
          "delegation1" : {
            "name" : "Microsoft.Databricks/workspaces",
            "actions" : [
              "Microsoft.Network/virtualNetworks/subnets/join/action",
              "Microsoft.Network/virtualNetworks/subnets/unprepareNetworkPolicies/action",
              "Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action"
            ]
          }
        }
      },
      "tags" : {}
    }
  }
}

resource "azurerm_subnet" "delegation_dynamic" {
  for_each             = local.subnets
  name                 = each.value.name
  resource_group_name  = azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = [each.value.cidr]
  dynamic "delegation" {
    for_each = contains(keys(each.value), "provider_specific") ? lookup(each.value.provider_specific, "delegations", {}) : {}
    content {
      name = delegation.key
      service_delegation {
        name    = delegation.value.name
        actions = sort(delegation.value.actions)
      }
    }
  }
  depends_on = [
    azurerm_virtual_network.example
  ]
}

// assign delegation static
resource "azurerm_subnet" "delegation_static1" {
  name                 = "subnet3"
  resource_group_name  = azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["192.168.3.0/24"]
  delegation {
    name = "delegation1"
    service_delegation {
      name    = "Microsoft.Web/serverFarms"
      actions = sort(["Microsoft.Network/virtualNetworks/subnets/action"])
    }
  }
}

resource "azurerm_subnet" "delegation_static2" {
  name                 = "subnet4"
  resource_group_name  = azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["192.168.4.0/24"]
  delegation {
    name = "delegation1"
    service_delegation {
      name    = "Microsoft.Databricks/workspaces"
      actions = sort(["Microsoft.Network/virtualNetworks/subnets/unprepareNetworkPolicies/action", "Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action", "Microsoft.Network/virtualNetworks/subnets/join/action"])
    }
  }
}

This code, with the removed and sorted delegations will lead to:

terraform plan
azurerm_resource_group.example: Refreshing state... [id=/subscriptions/73a9c577-8635-4b8f-80b7-072178ee4f2a/resourceGroups/delegation-example]
azurerm_virtual_network.example: Refreshing state... [id=/subscriptions/73a9c577-8635-4b8f-80b7-072178ee4f2a/resourceGroups/delegation-example/providers/Microsoft.Network/virtualNetworks/delegation-example]
azurerm_subnet.delegation_static2: Refreshing state... [id=/subscriptions/73a9c577-8635-4b8f-80b7-072178ee4f2a/resourceGroups/delegation-example/providers/Microsoft.Network/virtualNetworks/delegation-example/subnets/subnet4]
azurerm_subnet.delegation_dynamic["192.168.1.0/24"]: Refreshing state... [id=/subscriptions/73a9c577-8635-4b8f-80b7-072178ee4f2a/resourceGroups/delegation-example/providers/Microsoft.Network/virtualNetworks/delegation-example/subnets/subnet1]
azurerm_subnet.delegation_dynamic["192.168.2.0/24"]: Refreshing state... [id=/subscriptions/73a9c577-8635-4b8f-80b7-072178ee4f2a/resourceGroups/delegation-example/providers/Microsoft.Network/virtualNetworks/delegation-example/subnets/subnet2]
azurerm_subnet.delegation_static1: Refreshing state... [id=/subscriptions/73a9c577-8635-4b8f-80b7-072178ee4f2a/resourceGroups/delegation-example/providers/Microsoft.Network/virtualNetworks/delegation-example/subnets/subnet3]

No changes. Your infrastructure matches the configuration.

Is this caused by some hashing or where does this come from? Should this be somehow documented (anywhere else than here)? Should a fix be implemented, or should it just be documented?

Steps to Reproduce

Add and remove the sort function and see delegations get reapplied over and over again.

Of course with: terraform plan terraform apply

Important Factoids

No response

References

#4599 #4600

[neil-yechenwei]

Thanks for raising this issue. The elements on the actions are in alphabetical order. For now, please follow this rule or use ignore_changes to unblock you. Will submit a PR to fix it.

[jason-berk-k1x]

years later and still having the same issue....

this TF

delegation {
    name = "container_app_env_subnet_delegation"
    service_delegation {
      name = "Microsoft.App/environments"
      actions = [
        "Microsoft.Network/virtualNetworks/subnets/join/action",
        "Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action",
      ]
    }
  }

results in this on every apply

 ~ service_delegation {
              ~ actions = [
                    "Microsoft.Network/virtualNetworks/subnets/join/action",
                  + "Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action",
                ]
                name    = "Microsoft.App/environments"
            }

[ticiano-herrera-bzy]

thumbs up! still happening with provider 3.90.1

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.