terraform-provider-azurerm
terraform-provider-azurerm copied to clipboard
keyvault: support for creating items over a nested endpoint
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Terraform (and AzureRM Provider) Version
0.14.0 / 2.39.0
Affected Resource(s)
-
azurerm_key_vault_key
Terraform Configuration Files
provider "azurerm" {
version = "=2.39.0"
subscription_id = "5ab29627-1b06-47b1-bf94-251a9e6c74c7"
tenant_id = "90ad1900-2019-4ffa-b845-96f012d0dc5a"
skip_provider_registration = false
features {}
use_msi = true
}
provider "azurerm" {
version = "=2.39.0"
alias = "dnsprod"
subscription_id = "ed330c6f-5b25-4d9a-8db2-13a6b612c317"
tenant_id = "90ad1900-2019-4ffa-b845-96f012d0dc5a"
skip_provider_registration = true
features {}
use_msi = true
}
data "azurerm_client_config" "current" {}
data "azurerm_virtual_machine" "vm" {
name = "mangementvm"
resource_group_name = "development"
}
data "azurerm_subnet" "rg" {
name = "Storage"
virtual_network_name = "Policytest"
resource_group_name = "development"
}
resource "azurerm_resource_group" "rg" {
name = "TFtest2"
location = "uksouth"
}
resource "azurerm_key_vault" "main" {
name = "newkekvault215"
resource_group_name = azurerm_resource_group.rg.name
location = azurerm_resource_group.rg.location
tenant_id = data.azurerm_client_config.current.tenant_id
sku_name = "premium"
purge_protection_enabled = true
soft_delete_enabled = true
network_acls {
default_action = "Deny"
bypass = "AzureServices"
}
access_policy {
tenant_id = data.azurerm_client_config.current.tenant_id
object_id = data.azurerm_virtual_machine.vm.identity.0.principal_id
key_permissions = ["get", "create", "delete", "list", "restore", "recover", "unwrapkey", "wrapkey", "purge", "encrypt", "decrypt", "sign", "verify",]
}
}
data "azurerm_private_dns_zone" "key_vault_dns_private_zone" {
name = "privatelink.vaultcore.azure.net"
provider = azurerm.dnsprod
resource_group_name = "NewTestNetwork"
}
resource "azurerm_private_endpoint" "keyvault" {
name = "key_vault-terraform-endpoint"
location = azurerm_resource_group.rg.location
resource_group_name = azurerm_resource_group.rg.name
subnet_id = "${data.azurerm_subnet.rg.id}"
private_service_connection {
name = "key_vault-terraform-privateserviceconnection"
private_connection_resource_id = azurerm_key_vault.main.id
subresource_names = [ "vault" ]
is_manual_connection = false
}
private_dns_zone_group {
name = data.azurerm_private_dns_zone.key_vault_dns_private_zone.name
private_dns_zone_ids = [data.azurerm_private_dns_zone.key_vault_dns_private_zone.id]
}
}
resource "azurerm_key_vault_key" "key" {
name = "Storage-KEK200"
key_vault_id = azurerm_key_vault.main.id
key_type = "RSA-HSM"
key_size = 2048
expiration_date = "2050-01-01T00:00:00Z"
key_opts = [
"decrypt",
"encrypt",
"sign",
"unwrapKey",
"verify",
"wrapKey",
]
}
Debug Output
Panic Output
Expected Behaviour
The Key gets created.
Actual Behaviour
Error: Error checking for presence of existing Key "Storage-KEK200" (Key Vault "https://newkekvault215.vault.azure.net/"): keyvault.BaseClient#GetKey: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Client address is not authorized and caller is not a trusted service.\r\nClient address: 10.0.6.5 from unknown subnet\r\nCaller: appid=feb4f312-1ff0-4b05-94b0-933b72f199d5;oid=8257275c-a014-4140-ba4f-b3b656dc9f6d;iss=https://sts.windows.net/90ad1900-2019-4ffa-b845-96f012d0dc5a/;xms_mirid=/subscriptions//resourcegroups/development/providers/Microsoft.Compute/virtualMachines/mangementvm\r\nVault: newkekvault215;location=uksouth" InnerError={"code":"ForbiddenByFirewall"}
-
At this point the Private End Point, KV is created correctly and from the VM with the Managed Service Identity I can create a key with az cli for example over the Private End Point because the KV has no firewalls.
-
If without any change I do terraform apply again the key is deployed.
I've tried to put the Key Vault access policy via a separate resource and note this as a dependency for the key Vault Key resource and put in a timer delay but this didn't help.
Steps to Reproduce
-
terraform apply
Important Factoids
References
- #0000
Need to whitelist your Public IP (server for terraform administration) in the vault
Hi antanof we are all over Private, we orchestrate from a Private IP as you can see in the error message, we don't want to switch to a Public IP for orchestration, also we don't want a Public IP firewall rule in our KV', also note if we perform an Apply again after the initial Apply it works so fundamentally this is possible I believe, but some state related to the Private End Point is not being imported until you run terraform again so I'm sure a bug.
@cb900rr2000 , I would ask is it possible to achieve that in one terraform iteration at all instead of running terraform apply twice. In my case I am first using public MS API to create and configure kv + create private endpoints, once created I want to create keys using PE instead of public MS API and this is what I get:
Error: Error checking for presence of existing Key "default-database-key" (Key Vault "https://miswiercdemovault.vault.azure.net/"): keyvault.BaseClient#GetKey: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Client address is not authorized and caller is not a trusted service.\r\nClient address: <SOME_IP>\r\nCaller: appid=XXX;oid=YYY;iss=https://sts.windows.net/ZZZ/\r\nVault: miswiercdemovault;location=westeurope" InnerError={"code":"ForbiddenByFirewall"}
It works for me if I run terraform twice. Same like you I don't have FW.
You have to look at the parameters: allow or deny by default. If we deny except the Azure services, they will go through the public IP of the vault. Then you have the public ip and authorized subnets. I think you need to add the subnet of your administration vm to the keyvault firewall white list.
👋
At this time Terraform uses the Data Plane API to interact with Azure Key Vault for Certificates, Keys and Secrets - which is available over the public internet (although can be restricted using an IP Filter as described above). In order to support connecting to Key Vault over a Private Endpoint, we'd need to switch to using the Resource Manager API to do this, which may include permission changes - but this isn't supported at this time.
As such I've updated the issue title and tagged this as an enhancement to support creating Key Vault items over a Private Endpoint.
Thanks!
@tombuildsstuff Can you go into more detail for the work around? We are running into the error which I believe is caused by the same behavior?
Error: retrieving
contact` for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
Error: retrieving contact
for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
Error: retrieving contact
for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
##[error]Terraform command 'plan' failed with exit code '1'.: retrieving contact
for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded | retrieving contact
for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded | retrieving contact
for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
##[error]
Error: retrieving contact
for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
Error: retrieving contact
for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
Error: retrieving contact
for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
`
Have this issue as well but I don't understand. Using a private endpoint does not mean that you cannot reach it from the internet. OP choose to bypass AzureServices from network acl, so Azure Devops, where the terraform code often runs, could use Data Plane API just as if the key vault did not have a Private Endpoint.
network_acls {
default_action = "Deny"
bypass = "AzureServices"
}
👋
At this time Terraform uses the Data Plane API to interact with Azure Key Vault for Certificates, Keys and Secrets - which is available over the public internet (although can be restricted using an IP Filter as described above). In order to support connecting to Key Vault over a Private Endpoint, we'd need to switch to using the Resource Manager API to do this, which may include permission changes - but this isn't supported at this time.
> As such I've updated the issue title and tagged this as an enhancement to support creating Key Vault items over a Private Endpoint.
Thanks!
Hi @tombuildsstuff,
Can you provide any timeline on when this support could be in place?
Thanks!
Things that helped me workaround this issue until a fix is released: assigning certificate_permissions = [ "ManageContacts" ] to the Service Principal that controls terraform i also encountered context timeout, this was because i had private DNS for keyvault linked to the vnet and it prevented getting the public IP for the vault i was trying to modify. Removing the private DNS fixed the issue.
In order to support connecting to Key Vault over a Private Endpoint, we'd need to switch to using the Resource Manager API to do this, which may include permission changes - but this isn't supported at this time.
@tombuildsstuff do you have any pointers to this? I couldnt find any API docs around that. Thanks for any pointers!
Generally, I think the KeyVault implementation should have one option to turn off the data plane access (and disable those related minor feature if needed) Include: Do not ping the keyvault URL for checking existence. Do not make any API such as get contacts.
Making data plane access always cause additional trouble when we have private network endpoint.
At this time Terraform uses the Data Plane API to interact with Azure Key Vault for Certificates, Keys and Secrets - which is available over the public internet (although can be restricted using an IP Filter as described above). In order to support connecting to Key Vault over a Private Endpoint, we'd need to switch to using the Resource Manager API to do this, which may include permission changes - but this isn't supported at this time.
As such I've updated the issue title and tagged this as an enhancement to support creating Key Vault items over a Private Endpoint.
I am also getting this error (suddenly, too; to my knowledge, nothing has changed, and I can't find anything in the activity logs in Azure to indicate it).
But the KV for which I'm getting this on is set to "Allow access from: All networks" on the Networking / Firewalls and virtual networks page, and has no connections under "Private endpoint connections"…
Multiple issues have been closed as duplicates of this one. But I am not sur the following error is the exact same context as described here.
Error: retrieving contact for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded
We get this error when trying to refresh the state even if there are no data plane operations in the Terraform file, as described in may of the linked issues.
Unless you tell me that the access policy
is using the data plane?
We have a similar issue right now in our current environment the keyvault is created before peerings are created and state refresh before peering creation will fail due to the required dataplane access to set these values. We can workaround this issue by deploying the keyvault after peerings and DNS are done but that might cause other issues.
Agree with @oliviergaumond
@tombuildsstuff
Can you clarify if you are including all the problems mentioned in - https://github.com/hashicorp/terraform-provider-azurerm/issues/10501 within this issue now as well even though this issue is labeled very differently above?
Thanks!
Unfortunately latest Azure Terraform Provider doesn't solve the issue.
As of now I am still having troubles on this matter. I am pretty sure that my case has already been listed, but just in case: I had created the Azure key vault with the access policy and configured a private endpoint.
Until I removed the private endpoint manually (allowing public access didn't have an effect, nor deleting the policy in my case) Terraform wasn't able to refresh its state (failing with retrieving contact for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure
).
Once I removed the PE then Terraform was able to plan again and get the state for the key vault. Do we know of any workaround for the issue in the meanwhile, as the team looks for a solution?
Any progress on this issue in new year?
Any workaround?
In Azure DevOps I use an Azure DevOps Private Agent and ensure that it has access to the subnet where the key vault private endpoint resides. This is my work around. It works without issues.
What happens is that once Private Endpoints are enabled in the Key Vault, it shuts off public access. Terraform reaches out to the Key Vault to confirm that it exists and/or to make a change, etc... If the access from Terraform to the Key Vault is public, it will not be able to confirm access it thus it fails. I have seen a couple of different errors that are a symptom of the same issue.
I hope this helps someone.
I want to have an Azure Key Vault with this configuration:
- Public Access Denied (no public IP or VNet allowed)
- Private endpoint connected to a specific VNet
I'm executing the terraform commands from my laptop, this means that interaction towards Azure is done with a public IP.
First time that I create the Key Vault everything works smoothly, and the Azure Key Vault is created with associated private endpoint.
If I try to execute a new terraform plan
(without change nothing in terraform configuration), the execution stuck until the following error is thrown:
│ Error: retrieving `contact` for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context canceled
│
│ with module.infrastructure.module.key-vault.azurerm_key_vault.main,
│ on ../modules/key-vault/key-vault-main.tf line 2, in resource "azurerm_key_vault" "main":
│ 2: resource "azurerm_key_vault" "main" {
│
Following the @jpmicrosoft suggestion, I have tried to whitelist my IP. But this didn't solve problem. As second attempt I have tried to re-enable public access. Also this didn't solve problem.
The only way to solve it is to remove the private endpoint from Key Vault private endpoint connections section.
In this way terraform plan
restart to work, but, obviously, Key Vault won't have the associated private endpoint.
I want to have an Azure Key Vault with this configuration:
- Public Access Denied (no public IP or VNet allowed)
- Private endpoint connected to a specific VNet
I'm executing the terraform commands from my laptop, this means that interaction towards Azure is done with a public IP.
First time that I create the Key Vault everything works smoothly, and the Azure Key Vault is created with associated private endpoint.
If I try to execute a new
terraform plan
(without change nothing in terraform configuration), the execution stuck until the following error is thrown:│ Error: retrieving `contact` for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context canceled │ │ with module.infrastructure.module.key-vault.azurerm_key_vault.main, │ on ../modules/key-vault/key-vault-main.tf line 2, in resource "azurerm_key_vault" "main": │ 2: resource "azurerm_key_vault" "main" { │
Following the @jpmicrosoft suggestion, I have tried to whitelist my IP. But this didn't solve problem. As second attempt I have tried to re-enable public access. Also this didn't solve problem.
The only way to solve it is to remove the private endpoint from Key Vault private endpoint connections section. In this way
terraform plan
restart to work, but, obviously, Key Vault won't have the associated private endpoint.
@simonesavi The workaround is not to add IPs to the ACL but to ensure you are executing Terraform from a VM/Container/etc. that has access to the Private Endpoints subnet.
I hope this helps.
After a troubleshooting I can confirm that the problem was generated by a DNS misconfiguration which did not correctly resolve the key vault name.
Anyway you require access to the Private Endpoints subnet only if the key vault name is resolved with private IP.
If you are using a public DNS, asking for a name resolution, you will receive public IP and terraform plan
will works without problem.
I want to have an Azure Key Vault with this configuration:
- Public Access Denied (no public IP or VNet allowed)
- Private endpoint connected to a specific VNet
I'm executing the terraform commands from my laptop, this means that interaction towards Azure is done with a public IP. First time that I create the Key Vault everything works smoothly, and the Azure Key Vault is created with associated private endpoint. If I try to execute a new
terraform plan
(without change nothing in terraform configuration), the execution stuck until the following error is thrown:│ Error: retrieving `contact` for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context canceled │ │ with module.infrastructure.module.key-vault.azurerm_key_vault.main, │ on ../modules/key-vault/key-vault-main.tf line 2, in resource "azurerm_key_vault" "main": │ 2: resource "azurerm_key_vault" "main" { │
Following the @jpmicrosoft suggestion, I have tried to whitelist my IP. But this didn't solve problem. As second attempt I have tried to re-enable public access. Also this didn't solve problem. The only way to solve it is to remove the private endpoint from Key Vault private endpoint connections section. In this way
terraform plan
restart to work, but, obviously, Key Vault won't have the associated private endpoint.@simonesavi The workaround is not to add IPs to the ACL but to ensure you are executing Terraform from a VM/Container/etc. that has access to the Private Endpoints subnet.
I hope this helps.
Heya there, I have a VMSS with access to the subnets, however I can't get the private endpoint to deploy. We have azure policy's and networking in place to stop public access. So state is in a private endpoint and we can't use a public agent. The problem comes when the keyvault gets deployed and is set to deny, the terraform fails with waiting for the keyvault to become available. This then prevents the Private endpoint deploying. If I try to run it again, I get hit with the contact error mentioned because I'm trying to do dataplane access but I can't deploy the private endpoint in terraform because I can't refresh. Is this just impossible then or am I missing something obvious ?
@gabrielmccoll we solved a similar situation when using Private Build Agents with Key Vault and Private Endpoints by the use of a deployment timer:
resource "azurerm_key_vault" "stamp" {
name = "${local.prefix}-${local.location_short}-kv"
location = azurerm_resource_group.stamp.location
resource_group_name = azurerm_resource_group.stamp.name
tenant_id = data.azurerm_client_config.current.tenant_id
network_acls {
bypass = "None"
default_action = "Deny" # Deny all access - except for the private endpoint connections
}
sku_name = "standard"
}
# Give KV secret permissions to the service principal that runs the Terraform apply itself
resource "azurerm_key_vault_access_policy" "devops_pipeline_all" {
key_vault_id = azurerm_key_vault.stamp.id
tenant_id = data.azurerm_client_config.current.tenant_id
object_id = data.azurerm_client_config.current.object_id
secret_permissions = [
"Get", "List", "Delete", "Purge", "Set", "Backup", "Restore", "Recover"
]
}
resource "azurerm_private_endpoint" "buildagent_keyvault" {
name = "${local.prefix}-${local.location_short}-built-agent-keyvault-pe"
location = data.azurerm_resource_group.buildagent.location
resource_group_name = data.azurerm_resource_group.buildagent.name
subnet_id = "${data.azurerm_virtual_network.buildagent.id}/subnets/private-endpoints-snet"
private_dns_zone_group {
name = "privatednskeyvault"
private_dns_zone_ids = ["${data.azurerm_resource_group.buildagent.id}/providers/Microsoft.Network/privateDnsZones/privatelink.vaultcore.azure.net"]
}
private_service_connection {
name = "${local.prefix}-${local.location_short}-keyvault-buildagent-privateserviceconnection"
private_connection_resource_id = azurerm_key_vault.stamp.id
is_manual_connection = false
subresource_names = ["vault"]
}
}
resource "time_sleep" "wait_keyvault_pe" {
depends_on = [azurerm_private_endpoint.buildagent_keyvault]
create_duration = "300s" # 5min should give us enough time for the Private endpoint to come online
}
resource "azurerm_key_vault_secret" "secrets" {
# Every secret is depended on a) the access policy for the deploying service principal being created and b) - only when running in private mode - on the build agent private endpoint being up and running
depends_on = [azurerm_key_vault_access_policy.devops_pipeline_all, time_sleep.wait_keyvault_pe]
name = "foo"
value = "bar"
key_vault_id = azurerm_key_vault.stamp.id
}
@gabrielmccoll we solved a similar situation when using Private Build Agents with Key Vault and Private Endpoints by the use of a deployment timer:
resource "azurerm_key_vault" "stamp" { name = "${local.prefix}-${local.location_short}-kv" location = azurerm_resource_group.stamp.location resource_group_name = azurerm_resource_group.stamp.name tenant_id = data.azurerm_client_config.current.tenant_id network_acls { bypass = "None" default_action = "Deny" # Deny all access - except for the private endpoint connections } sku_name = "standard" } # Give KV secret permissions to the service principal that runs the Terraform apply itself resource "azurerm_key_vault_access_policy" "devops_pipeline_all" { key_vault_id = azurerm_key_vault.stamp.id tenant_id = data.azurerm_client_config.current.tenant_id object_id = data.azurerm_client_config.current.object_id secret_permissions = [ "Get", "List", "Delete", "Purge", "Set", "Backup", "Restore", "Recover" ] } resource "azurerm_private_endpoint" "buildagent_keyvault" { name = "${local.prefix}-${local.location_short}-built-agent-keyvault-pe" location = data.azurerm_resource_group.buildagent.location resource_group_name = data.azurerm_resource_group.buildagent.name subnet_id = "${data.azurerm_virtual_network.buildagent.id}/subnets/private-endpoints-snet" private_dns_zone_group { name = "privatednskeyvault" private_dns_zone_ids = ["${data.azurerm_resource_group.buildagent.id}/providers/Microsoft.Network/privateDnsZones/privatelink.vaultcore.azure.net"] } private_service_connection { name = "${local.prefix}-${local.location_short}-keyvault-buildagent-privateserviceconnection" private_connection_resource_id = azurerm_key_vault.stamp.id is_manual_connection = false subresource_names = ["vault"] } } resource "time_sleep" "wait_keyvault_pe" { depends_on = [azurerm_private_endpoint.buildagent_keyvault] create_duration = "300s" # 5min should give us enough time for the Private endpoint to come online } resource "azurerm_key_vault_secret" "secrets" { # Every secret is depended on a) the access policy for the deploying service principal being created and b) - only when running in private mode - on the build agent private endpoint being up and running depends_on = [azurerm_key_vault_access_policy.devops_pipeline_all, time_sleep.wait_keyvault_pe] name = "foo" value = "bar" key_vault_id = azurerm_key_vault.stamp.id }
hey there, thank you for this, I've given it a try but I get same error "##[error]╷
│ Error: retrieving contact
for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded"
Basically I can't get public routing to the data planeaccess due to policies and networking so I have to do it all private routing but that seems impossible since I can't get the private endpoint to deploy due to the above error, soon as the keyvault deploys it's a closed book it seems.
make sure the key vault gets newly deployed. The timer only applies the first time when the KV is actually getting created. On the next runs the timer should not be required anymore as then the Private Endpoint is already there and is used by your private build agent
make sure the key vault gets newly deployed. The timer only applies the first time when the KV is actually getting created. On the next runs the timer should not be required anymore as then the Private Endpoint is already there and is used by your private build agent
thanks again for the reply and help, the key vault is getting freshly deployed but doesn't complete as it failed with the above error. The private endpoint doesn't deploy because the initial deployment of the keyvault doesn't get all the way there it seems. So the PE never exists to be used for the next run.
looks like somewhere in your TF state you still have references to some KV-child resources (certificates?!). TF tries to check their state I guess. Try to start from a completely fresh (clean) state file
I am able to pull the secrets through public interface(coresvcazvault01.vault.azure.net). I have created a private endpoint and configured the private dns zone and hostname with "coresvcazvault01.privatelink.vaultcore.azure.net". Anyone please help me on how to configure terraform resource to use this private endpoint to pull the secrets from the Azure Key vault. Would like to avoid using public interface for security reason.
data "azurerm_key_vault" "vault" {
name = "coresvcazvault01"
resource_group_name = "azkeyvaulttest01"
}
data "azurerm_key_vault_secret" "secret1" {
name = "secret1"
key_vault_id = data.azurerm_key_vault.vault.id
}
output "secret_value" {
value = data.azurerm_key_vault_secret.secret1.value
sensitive = true
}
output "vault_url1" {
value = data.azurerm_key_vault.vault.vault_uri
sensitive = false
}
root@terraform# terraform output
secret_value = <sensitive>
vault_url1 = "https://coresvcazvault01.vault.azure.net/"