terraform-provider-azurerm
terraform-provider-azurerm copied to clipboard
hashicorp
Example of using the Subnet Association resources with Azure Policy
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Description
Larger organizations utilize Azure Policies to ensure that a Subnets contain a Network Security Group/Route Table ID at creation time, which is incompatible with the azurerm_subnet_network_security_group_association and azurerm_subnet_route_table_association resources - which are required to be able to workaround issues in the Azure API during terraform destroy.
Unfortunately these issues are unavoidable due to the design of the Azure API - and whilst we could seek to reintroduce these two fields - ultimately this'd reintroduce the issue and so isn't a viable solution.
These "association" resources create an empty Subnet and then subsequently patch them to add the network_security_group_id and the route_table_id fields - but crucially by having these split out allow operations to be ordered such that these destroy operations aren't possible.
One option to work around this is to use the azurerm_virtual_network resource to define the subnets with these fields inline - however this only works when these subnets are defined centrally and so doesn't work for all users/scenarios.
As such this issue covers adding an example Azure Policy to achieve this scenario, namely:
- Allowing the creation of a Subnet (with no Network Security Group ID / Route Table ID when there's no Network Interfaces [IP Configurations] present)
- When attempting to create a Network Interface [IP Configuration] within the Subnet, require that a Network Security Group ID/Route Table ID exists
This Policy should allow organizations to comply with Policy requirements (requiring that all Subnets have a Network Security Group/Route Table ID) whilst working around the issues in the Azure API.
New or Affected Resource(s)
-
azurerm_policy_definition -
azurerm_subnet -
azurerm_subnet_network_security_group_association -
azurerm_subnet_route_table_association
References
- #7196
- #7586
- #7987
- #8261
I am also having this issue.... it's quite blocking as well here.
I now create a VNET through this code:
resource "azurerm_virtual_network" "vnet_hub" {
name = "vnet-hub"
location = azurerm_resource_group.rg_connectivity_hub.location
resource_group_name = azurerm_resource_group.rg_connectivity_hub.name
address_space = ["10.0.0.0/16"]
tags = {
environment = "global"
}
subnet {
name = "GatewaySubnet" # Required name, do not change
address_prefix = "10.0.255.224/27"
security_group = azurerm_network_security_group.nsg_hub_gateway.id
}
subnet {
name = "s-hub-mgmt"
address_prefix = "10.0.0.64/27"
security_group = azurerm_network_security_group.nsg_hub_mgmt.id
}
subnet {
name = "s-hub-dmz"
address_prefix = "10.0.0.32/27"
security_group = azurerm_network_security_group.nsg_hub_dmz.id
}
}
However the main issue is that while I expect this to create the subnets, I am not able to fetch the IDs of the subnet (even though they should get exported as stated by https://github.com/terraform-providers/terraform-provider-azurerm/pull/1913). Checking into the code https://github.com/terraform-providers/terraform-provider-azurerm/blob/master/azurerm/internal/services/network/virtual_network_resource.go#L423 provides that I should be able to set subnet_id = values(azurerm_virtual_network.vnet_hub.subnet)[0].id to then initialize the gateway. This however is not working either and returns that the Set is empty.
Update: Fixed I can deploy now with the policy, below the config I used:
# Network Security Groups (NSGs)
resource "azurerm_network_security_group" "nsg_hub_mgmt" {
name = "nsg-hub-mgmt"
location = azurerm_resource_group.rg_connectivity_hub.location
resource_group_name = azurerm_resource_group.rg_connectivity_hub.name
}
resource "azurerm_network_security_group" "nsg_hub_dmz" {
name = "nsg-hub-dmz"
location = azurerm_resource_group.rg_connectivity_hub.location
resource_group_name = azurerm_resource_group.rg_connectivity_hub.name
}
# Virtual Network (vNET)
resource "azurerm_virtual_network" "vnet_hub" {
name = "vnet-hub"
location = azurerm_resource_group.rg_connectivity_hub.location
resource_group_name = azurerm_resource_group.rg_connectivity_hub.name
address_space = ["10.0.0.0/16"]
tags = {
environment = "global"
}
# Our Gateway Subnet, it cannot have a NSG!
subnet {
name = "GatewaySubnet" # Required name, do not change
address_prefix = "10.0.255.224/27"
}
subnet {
name = "s-hub-mgmt"
address_prefix = "10.0.0.64/27"
security_group = azurerm_network_security_group.nsg_hub_mgmt.id
}
subnet {
name = "s-hub-dmz"
address_prefix = "10.0.0.32/27"
security_group = azurerm_network_security_group.nsg_hub_dmz.id
}
}
this only works when these subnets are defined centrally and so doesn't work for all users/scenarios
Do you have an example of how this can be achieved if the subnets aren't defined centrally? This is a real blocker for us.
You have to change the azure policy if you aren't defining the vnets and subnets all at once.
@tombuildsstuff when you talk about "When attempting to create a Network Interface [IP Configuration] within the Subnet, require that a Network Security Group ID/Route Table ID exists" are you referring to the NSG created at the Network Interface or at the Subnet?
Hi @courageDeveloper, @Asos-RiverPhillips & @XavierGeerinck,
You can reference the subnet attributes as part of the VNET resource block like the below:
output "subnet-id" {
value = azurerm_virtual_network.vnet.subnet.*.id[0]
}
output "subnet-name" {
value = azurerm_virtual_network.vnet.subnet.*.name[0]
}
Where "0" is the first subnet block defined, so GatewaySubnet in the above example.
Hope that helps
I don't see mentions of service_endpoints, but not PrivateLink Endpoints.
Inline subnet does not support attributes enforce_private_link_endpoint_network_policies or enforce_private_link_service_network_policies and with Azure Policies in place, inline is the only method to create subnets - therefore subnets cannot be created that support Private Link Services or Endpoints.
We're in an environment where we do not have the luxury of disabling or changing Azure Policies, so that approach is not an option open to us.
I am facing the same problem as @hobti01. More detailed description here: https://discuss.hashicorp.com/t/circumvent-azure-policy-deny-subnet-without-nsg/32422
As far as I'm aware there are two options:
- Create the subnet/nsg as an all-in-one resource in terraform
- Disable the Azure Policy
If both are unavailable to you then you're at a dead-end because I do not believe there is any other feasible way to do this
@tombuildsstuff I've tried to play with inline NSG association by reintroducing the security_group to azurerm_subnet in https://github.com/weisdd/terraform-provider-azurerm/commit/8ae1cdb66a43e341ebeee1d798c3168a9503e206 , and haven't seen any errors during multiple apply-destroy manual tests. Could you, please, share an example of code from the past that would more or less consistently reproduce the issue you're referring to in the first message? (https://github.com/hashicorp/terraform-provider-azurerm/issues/9022#issue-729540778) Since it's been over 1.5 years, maybe something has improved on the Azure side. I'd love to test it all.
Thanks!
(Upd): Here's the example I'm using:
provider "azurerm" {
features {}
}
resource "azurerm_resource_group" "example" {
name = "example-resources"
location = "West Europe"
}
resource "azurerm_virtual_network" "example" {
name = "example-network"
address_space = ["10.0.0.0/16"]
location = azurerm_resource_group.example.location
resource_group_name = azurerm_resource_group.example.name
}
resource "azurerm_subnet" "example" {
name = "frontend"
resource_group_name = azurerm_resource_group.example.name
virtual_network_name = azurerm_virtual_network.example.name
address_prefixes = ["10.0.2.0/24"]
security_group = azurerm_network_security_group.example.id
}
resource "azurerm_network_security_group" "example" {
name = "example-nsg"
location = azurerm_resource_group.example.location
resource_group_name = azurerm_resource_group.example.name
security_rule {
name = "test123"
priority = 100
direction = "Inbound"
access = "Allow"
protocol = "Tcp"
source_port_range = "*"
destination_port_range = "*"
source_address_prefix = "*"
destination_address_prefix = "*"
}
}
$ terraform apply
[...]
azurerm_resource_group.example: Creating...
azurerm_resource_group.example: Creation complete after 1s [id=/subscriptions/64842ced-4781-416f-81ff-482b7f562581/resourceGroups/example-resources]
azurerm_virtual_network.example: Creating...
azurerm_network_security_group.example: Creating...
azurerm_virtual_network.example: Creation complete after 4s [id=/subscriptions/64842ced-4781-416f-81ff-482b7f562581/resourceGroups/example-resources/providers/Microsoft.Network/virtualNetworks/example-network]
azurerm_network_security_group.example: Creation complete after 4s [id=/subscriptions/64842ced-4781-416f-81ff-482b7f562581/resourceGroups/example-resources/providers/Microsoft.Network/networkSecurityGroups/example-nsg]
azurerm_subnet.example: Creating...
azurerm_subnet.example: Creation complete after 4s [id=/subscriptions/64842ced-4781-416f-81ff-482b7f562581/resourceGroups/example-resources/providers/Microsoft.Network/virtualNetworks/example-network/subnets/frontend]
Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
$ terraform destroy
[...]
azurerm_subnet.example: Destroying... [id=/subscriptions/64842ced-4781-416f-81ff-482b7f562581/resourceGroups/example-resources/providers/Microsoft.Network/virtualNetworks/example-network/subnets/frontend]
azurerm_subnet.example: Still destroying... [id=/subscriptions/64842ced-4781-416f-81ff-...works/example-network/subnets/frontend, 10s elapsed]
azurerm_subnet.example: Destruction complete after 11s
azurerm_virtual_network.example: Destroying... [id=/subscriptions/64842ced-4781-416f-81ff-482b7f562581/resourceGroups/example-resources/providers/Microsoft.Network/virtualNetworks/example-network]
azurerm_network_security_group.example: Destroying... [id=/subscriptions/64842ced-4781-416f-81ff-482b7f562581/resourceGroups/example-resources/providers/Microsoft.Network/networkSecurityGroups/example-nsg]
azurerm_virtual_network.example: Still destroying... [id=/subscriptions/64842ced-4781-416f-81ff-...etwork/virtualNetworks/example-network, 10s elapsed]
azurerm_network_security_group.example: Still destroying... [id=/subscriptions/64842ced-4781-416f-81ff-...work/networkSecurityGroups/example-nsg, 10s elapsed]
azurerm_virtual_network.example: Destruction complete after 11s
azurerm_network_security_group.example: Destruction complete after 11s
azurerm_resource_group.example: Destroying... [id=/subscriptions/64842ced-4781-416f-81ff-482b7f562581/resourceGroups/example-resources]
azurerm_resource_group.example: Still destroying... [id=/subscriptions/64842ced-4781-416f-81ff-...62581/resourceGroups/example-resources, 10s elapsed]
azurerm_resource_group.example: Destruction complete after 15s
Destroy complete! Resources: 4 destroyed.
Upd2: I saw your comment in https://github.com/hashicorp/terraform-provider-azurerm/issues/3653#issuecomment-501995196, which said:
"There's a bunch of reasons for this (mostly user experience related), but the main one is to work around a circular reference in Azure, where it's possible to create resources in any order; but on deletion you encounter an error where the Subnet deletion will fail unless the Network Security Group is detached first."
When I was doing my tests, there was a policy in place that would prohibit me to create a subnet without an NSG attached or to detach the NSG from the subnet after creation. Thus, it seems it's now possible to remove a subnet without detaching NSG first.
@tombuildsstuff to make sure the things I observed (shared in the previous comment) are not temporary, I opened a ticket with Microsoft from our corporate account:
(Question): "[...] I've tried to reintroduce NSG reference back to the
azurerm_subnetresource (https://github.com/weisdd/terraform-provider-azurerm/commit/8ae1cdb66a43e341ebeee1d798c3168a9503e206) and found out that now it's actually possible to delete a subnet without detaching NSG, so Hashicorp's complaint seems to be no longer relevant. Could you track, when did that change happen in the Azure API backend? Will that behaviour persist over time? [...]" (Answer): "I confirm that the logic that allow subnet deletion without detaching NSG or UDR is permanent."
Unfortunately, I'm not aware about the exact date when this change was introduced yet, still trying to request that info from Microsoft.
Given this information, would Hashicorp consider reintroducing the binding logic back to the subnet resource? :) I understand that you've been flooded with similar requests, though I haven't seen anyone saying that the Azure backend logic has got fixed at some point, thus everything is likely to work flawlessly now. I can prepare a PR with tests if needed.
Since we have contracts with both Hashicorp and Microsoft, we can try to organise an internal email thread with all parties involved if there is any concern.
Thanks!
@weisdd unfortunately reintroducing these fields to the azurerm_subnet resource would end up reintroduce the (hard to debug, but common in complex configurations) issues which caused this - so we wouldn't accept a PR to reintroduce those fields.
If the Subnet's Delete API has changed it's behaviour then we should look to update the Delete method in the azurerm_subnet resource to account for this (checking and raising an error, if there's an association found) - but since that's a separate topic to this issue would you mind opening a new feature request for that?
Thanks!
Our scenario is the same as what Tom mentions. A default UDR is applied via policy - so when I tried to apply the association I would get that a resource already exists and I had to import it.
After being on a call yesterday with @grayzu I was pointed to azapi_update provider to test out a possible solution. For my needs I call this success! I create my subnet and nsg association with azurerm provider and the UDR association with azapi_update provider:
resource "azapi_update_resource" "subnet-asp-udr" {
type = "Microsoft.Network/virtualNetworks/subnets@2020-11-01"
resource_id = azurerm_subnet.asp.id
depends_on = [
azurerm_subnet.asp
]
body = jsonencode({
properties = {
routeTable = {
id = local.udrdefaultid
}
}
})
}
I even tested it for configuration drift (I changed the UDR from the portal manually) and it was picked up by terraform plan.
Please add this to a list of workarounds.
@sioakim I might misunderstand something, though how is that different from using azurerm_subnet_route_table_association directly? I think the call to modify the subnet properties is made after subnet creation, so if there's a blocking (Deny) policy in place, terraform will never execute azapi_update_resource, because it'll fail earlier, at the subnet creation step.
@weisdd
As per initial post:
Larger organizations utilize Azure Policies to ensure that a Subnets contain a Network Security Group/Route Table ID at creation time
For me this translates to having a policy that will add the subnet association and/or the NSG association on creation. Note you can not specify the associations from Terraform on creation. You need to create them as a second step. The problem is that if the policy has added an association already your 2nd step in terraform (that is specifying the association) will fail. Our policy doesn’t block Update. It just wants to make sure that you don’t have a blank value for the UDR.
`resource "azapi_resource" "cosmos_vnet" { # issue placeholder type = "Microsoft.Network/virtualNetworks@2022-05-01" name = <> parent_id = <resource_group_id> location = <>
depends_on = [ ]
body = jsonencode({
properties = {
addressSpace = {
addressPrefixes = <>
}
subnets = [
{
name = <>
"properties" : {
addressPrefix = <>
networkSecurityGroup = {
id = <azurerm_network_security_group.exmample.id>
}
routeTable = {
id = <azurerm_route_table.example.id>
}
}
type = "Microsoft.Network/virtualNetworks/subnets"
},
{
name : <>,
"properties" : {
addressPrefix = <>
networkSecurityGroup = {
id = <azurerm_network_security_group.example.id>
}
},
type : "Microsoft.Network/virtualNetworks/subnets"
}
]
}
})
}`
This solution seems to be working for us, when policies are introduced. But the use of azapi_resource introduces a dependency between azurerm and azureapi. Did somebody find a solution that uses solely azurerm?
UPDATE: apologies I made an error. The terraform issue still exists - however for the other two methods I was getting the policy violation because I was not using a specific route table (not mentioned in the error). When I switched to it the azapi update and azcli both worked
I also have this with azurerm_subnet + azurerm_subnet_route_table_association. I dont own the vnet, and tried az cli and azapi below. It appears there is no option but to manually create the subnet in the portal (which works)
az network vnet subnet create ... --route-table... fails.
as does
resource "azapi_update_resource" "add_bridge_subnet" {
type = "Microsoft.Network/virtualNetworks@2022-05-01"
resource_id = data.azurerm_virtual_network.protected_vnet.id
body = jsonencode({
properties = {
subnets = [
{
name = "${var.resource_prefix}-sn"
"properties" : {
addressPrefix = var.bridge_subnet_cidr
routeTable = {
id = data.azurerm_route_table.spoke_rt.id
}
}
type = "Microsoft.Network/virtualNetworks/subnets"
}
]
}
})
depends_on = [
data.azurerm_route_table.spoke_rt,
data.azurerm_virtual_network.protected_vnet,
]
}
│ "error": {
│ "code": "RequestDisallowedByPolicy",
│ "message": "Resource N was disallowed by policy. Policy identifiers.........Enforce a RouteTable on every subnet....
A bit of a side-note, but still related to the same policy that's being discussed here: AKS clusters with kubenet CNI (=managed vnet) can now be deleted (with no extra actions) despite the policy in place, it works for both Azure Portal and azurerm provider:
The current plan is to still detach the NSG before deleting the subnet (it handles some useful edge cases for us, so we don't want to remove it), but to make the deletion best-effort - if it does fail due to policy or some other reason, we'll proceed with cleaning everything up and deleting the node resource group. We'll log the error on our side for support but not return a failure state to you unless we're actually unable to delete everything.
https://github.com/Azure/AKS/issues/3111#issuecomment-1460254365
If you're hit with this limitation and need to create subnets with configuration like delegation, service endpoint and private endpoint support you can't use the solution within azurerm_virtual_network resource.
I posted an example in the following issue on how to solve it using the azapi_resource: https://github.com/hashicorp/terraform-provider-azurerm/issues/3917#issuecomment-1550246929
Full documentation can be found here: Microsoft.Network virtualNetworks/subnets
Update: Fixed I can deploy now with the policy, below the config I used:
# Network Security Groups (NSGs) resource "azurerm_network_security_group" "nsg_hub_mgmt" { name = "nsg-hub-mgmt" location = azurerm_resource_group.rg_connectivity_hub.location resource_group_name = azurerm_resource_group.rg_connectivity_hub.name } resource "azurerm_network_security_group" "nsg_hub_dmz" { name = "nsg-hub-dmz" location = azurerm_resource_group.rg_connectivity_hub.location resource_group_name = azurerm_resource_group.rg_connectivity_hub.name } # Virtual Network (vNET) resource "azurerm_virtual_network" "vnet_hub" { name = "vnet-hub" location = azurerm_resource_group.rg_connectivity_hub.location resource_group_name = azurerm_resource_group.rg_connectivity_hub.name address_space = ["10.0.0.0/16"] tags = { environment = "global" } # Our Gateway Subnet, it cannot have a NSG! subnet { name = "GatewaySubnet" # Required name, do not change address_prefix = "10.0.255.224/27" } subnet { name = "s-hub-mgmt" address_prefix = "10.0.0.64/27" security_group = azurerm_network_security_group.nsg_hub_mgmt.id } subnet { name = "s-hub-dmz" address_prefix = "10.0.0.32/27" security_group = azurerm_network_security_group.nsg_hub_dmz.id } }
This does not seem to work anymore (Terraform v1.5.1 on windows_386 + provider registry.terraform.io/hashicorp/azurerm v3.63.0). When I try to create a vnet with embedded subnet, I am getting the following error message:
╷ │ Error: Incorrect attribute value type │ │ on dev-ops-agents.tf line 22, in resource "azurerm_virtual_network" "default": │ 22: subnet = [ │ 23: { │ 24: address_prefix = "10.0.0.0/24" │ 25: name = "default" │ 26: security_group = azurerm_network_security_group.default.id │ 27: } │ 28: ] │ │ Inappropriate value for attribute "subnet": element 0: attribute "id" is required. ╵
Here is the full code:
provider "azurerm" {
features {}
subscription_id = "${var.subscriptionId}"
}
resource "azurerm_resource_group" "default" {
name = "vmssagents"
location = "West Europe"
tags = {
environment = "azuredevops"
}
}
resource "azurerm_virtual_network" "default" {
name = "vmssagents-vnet"
address_space = ["10.0.0.0/16"]
location = azurerm_resource_group.default.location
resource_group_name = azurerm_resource_group.default.name
subnet = [
{
address_prefix = "10.0.0.0/24"
name = "default"
security_group = azurerm_network_security_group.default.id
}
]
tags = {
environment = "azuredevops"
}
}
resource "azurerm_network_security_group" "default" {
name = "vmssagents-vnet-nsg"
location = azurerm_resource_group.default.location
resource_group_name = azurerm_resource_group.default.name
tags = {
environment = "azuredevops"
}
}
Why does it expect me to deliver the ID of a resource that I am asking it to create?
Please help. Thanks!
@thesse1 you need to specify each subnet in a separate block whereas you're using an array:
subnet = [
{
address_prefix = "10.0.0.0/24"
name = "default"
security_group = azurerm_network_security_group.default.id
}
]
=>
subnet {
address_prefix = "10.0.0.0/24"
name = "default"
security_group = azurerm_network_security_group.default.id
}
Here is another great workaround to mitigate the policy issue, I created a terraform module that uses the Azure/azapi resource to be able to create subnets that support delegation, service endpoints, as well as use both existing NSG and Route Tables, + is also able to create NSG and Route Table in one go,
It should help until an official resolution within azurerm is in place.
Terraform Registry: haflidif/alz-subnet/azurerm Github Repo: terraform-azurerm-alz-subnet
Has the policy example been created for this issue? I use "Azure/caf-enterprise-scale/azurerm" for deploying LZ for Azure. Constantly disabling or adding exemptions on policy whenever an issue occurs like this is not really a solution...
encountered the same issue, unless the subnet interface is fully replicated under the vnet resource, this will be an open issue. eg delegations
Hi @tombuildsstuff, you mentioned in your initial post that
As such this issue covers adding an example Azure Policy to achieve this scenario, namely:
Allowing the creation of a Subnet (with no Network Security Group ID / Route Table ID when there's no Network Interfaces [IP Configurations] present)
When attempting to create a Network Interface [IP Configuration] within the Subnet, require that a Network Security Group ID/Route Table ID exists
After some testing we are pretty sure that it is not possible to achieve this with Azure policies, although I hope we have missed something here. The first part is easy, e.g. just use no policy for subnet creation. But for the second part, the policy would deny the creation of the network interface (e.g. work on the network interface object) and I don't see any way to check the properties of the subnet object (whether a NSG is associated with the subnet or not) because the policy operates on the network interface and not the subnet.
Anyway, I hope we are missing something here, so I hope someone has an example of a working policy that implements this setup.
