Azure KeyVault Error: Provider produced inconsistent result after apply

[GregBillings]

trafficstars

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.14.8 AzureRM: terraform-provider-azurerm_v2.52.0_x5

Affected Resource(s)

• azurerm_key_vault_secret

Terraform Configuration Files

resource "azurerm_key_vault_secret" "mysecretvalue" {
  name         = "secretvaluename"
  value        = var.some_value_from_var
  key_vault_id = data.terraform_remote_state.remote_terraform_cloud_state.outputs.key_vault_id
}

Debug Output

Error: Provider produced inconsistent result after apply

When applying changes to azurerm_key_vault_secret.mysecretvalue, provider "registry.terraform.io/hashicorp/azurerm" produced an unexpected new value: Root resource was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own issue tracker.

Expected Behaviour

secret added to the keyvault

Actual Behaviour

The keyvault secret was added to the keyvault with the correct value, but the terraform apply failed with the error above. When re-running again, the new error is that the value already exists but isn't tracked in the terraform state

Steps to Reproduce

1. Create a terraform script that creates an azure keyvault and then outputs the ID as an output variable
2. Create another terraform script with a different remote state that pulls in the first remote state via:

data "terraform_remote_state" "remotestate" {
  backend = "remote"

  config = {
    organization = "my-org"
    workspaces = {
      name = "first-remote-state"
    }
  }
}

3. Attempt to create a new keyvault secret using the id from the output of the first remote state:

resource "azurerm_key_vault_secret" "mysecretvalue" {
  name         = "MySecretValue"
  value        = var.some_value_from_var
  key_vault_id = data.terraform_remote_state.remotestate.outputs.key_vault_id
}

References

• #10227

[KillianW]

This Issue is plaguing my pipelines at the moment. Is it possibly related to a similar underlying caching issue as #10602 ?

[dkirrane]

I had raised this previously here https://github.com/terraform-providers/terraform-provider-azurerm/issues/10227. Still seeing it with Terraform v0.14.9 & hashicorp/azurerm 2.56.0

If I re-run Terrafrom plan/apply I hit

Error: A resource with the ID "https://my-kv.vault.azure.net/secrets/my-secret/12345b12345cd4cd18c12345edb3c3cd" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_key_vault_secret" for more information.

[kousourakis]

Have the same issue with azurerm v2.71 terraform 1.0.4

deleted the remote state and recreated from scratch and it worked fine. I previously upgraded from 2.70 -> 2.71 and terraform 1.0.1 -> 1.0.4

[sinbai]

@GregBillings I am investigating this issue, but I can't repro it by following the steps below. Could you help confirm whether the following steps can repro this issue on your side? In addition, I would like to confirm whether you have done other operations before doing the following? Any detail information is greatly appreciated.

Steps to Reproduce

1. Create two new terraform cloud workspaces(ReproBug and ReproBug1).
2. Creates an azure keyvault on ReproBug workspace and then outputs the ID as an output variable via:

 terraform {
      required_providers {
        azurerm = {
          source = "hashicorp/azurerm"
         version = "2.52.0"
        }
      }
    
       backend "remote" {
       organization = "my-org-elena"
        workspaces {
          name = "ReproBug"
        }
     }
    }
    
    provider "azurerm" {
      features {}
    }
    
    resource "azurerm_resource_group" "test" {
      name     = "myTFResourceGroup"
      location = "westus2"
    }
    
    resource "azurerm_key_vault" "test" {
      name                       = "acctestkv-elena05"
      location                   = azurerm_resource_group.test.location
      resource_group_name        = azurerm_resource_group.test.name
      tenant_id                  = "XXXXXXXXX"
      sku_name                   = "standard"
      soft_delete_retention_days = 7
    
      access_policy {
        tenant_id = "XXXXXXX"
        object_id = "XXXXXXX"
    
        key_permissions = [
          "Get",
         "Delete",
        ]
    
        secret_permissions = [
          "Get",
          "Delete",
          "List",
          "Purge",
          "Recover",
          "Set",
        ]
      }
    
      tags = {
        environment = "Production"
      }
    }
    
    output "key_vault_id" {
      value = azurerm_key_vault.test.id
    }

3. create a new keyvault secret on ReproBug1 workspace using the id from the output of the first remote state via:

terraform {
      required_providers {
        azurerm = {
          source = "hashicorp/azurerm"
         version = "2.52.0"
        }
      }
    
       backend "remote" {
       organization = "my-org-elena"
        workspaces {
          name = "ReproBug1"
        }
     }
    }
    
    provider "azurerm" {
      features {}
    }
    
    data "terraform_remote_state" "remotestate" {
      backend = "remote"
    
      config = {
        organization = "my-org-elena"
        workspaces = {
          name = "ReproBug"
        }
      }
    }
    
    resource "azurerm_key_vault_secret" "mysecretvalue" {
      name         = "MySecretValue2"
      value        = "test"
      key_vault_id = data.terraform_remote_state.remotestate.outputs.key_vault_id
    }

Note: Remote state is managed by Terraform Cloud, issue can't be reproduced when Execution Mode is remote or local.

My Terraform (and AzureRM Provider) Version

[sinbai]

Have the same issue with azurerm v2.71 terraform 1.0.4

deleted the remote state and recreated from scratch and it worked fine. I previously upgraded from 2.70 -> 2.71 and terraform 1.0.1 -> 1.0.4

@elthanor I can't repro this issue, below is my repro steps. Could you provide more details to help me repro this issue? 1.Create two new terraform cloud workspaces(ReproBug1 and ReproBug2). 2.Create keyvault with azurerm v2.70 and terraform 1.0.1 on ReproBug1 workspace. 3.Update to azurerm provider to v2.7.1 and terraform to 1.0.4, add a new secret to the existing keyvault on workspace ReproBug2.

Note: Remote state is managed by Terraform Cloud, issue can't be reproduced when Execution Mode is remote or local.

[mybayern1974]

I would like to Re sinbai's finding that I cannot repro this issue either. The app I used is TF v1.0.5 and AzureRM provider v2.74 (this is the latest Azure provider version when I'm typing now). What I tried is creating a KV + KVSec first, and then use that created KV as a data source and create another KVSec. All the above was done by TF and all of them worked well w/o getting any command line error msg. I admit I just tried them w/ local state rather than remote state.

With noticing there are 20+ thump up to this issue, I do believe people ran into issues as described here. While to serve my or sinbai's troubleshooting, can anyone here provide more contexts in terms of step-by-step-repro-this-issue?

In addition, it would be helpful if below info can be provided:

1. Does this issue only repro when using remote state rather than local state?
2. Does this issue stable repro or happen intermittently?
3. For people who ran into this issue, do you use TF to manage all things (KV, KVSec, etc.) rather than using any other client tools (portal, CLI, etc) to co-manage resources?
4. For people who ran into this issue, is it possible there were someone-else/some-other-client-tooling manipulating the same resource (KV) at the same time when you used TF to manage that? If so, might this symptom be a result of conflict-manipulation-against-the-same-KV?

[MaksymChornyi]

@mybayern1974 Terraform 1.0.0 azurerm 2.71.0

1. I use backend "azurerm"
2. The problem occurs in ~70% of attempts
3. KeyVault has been created separately with the local state and I don't have this issue with the local state
4. We don't use KeyVault at the same moment

[futureviperowner]

When I ran into this issue, I was using a data resource to look up information about another key vault that was created outside of my terraform scripts. I was using the resource ID returned by the data source to create additional azurerm_key_vault_secret resources in that vault.

[mybayern1974]

@viper4u , I still could not repro after using remote states (use backend "azurerm") and using TF/AzureRM version you specified 😕. With seeing you mentioned 70% repro rate, I ran things 10 times and all of them succeeded.

Below are my repro steps with .tf config

1. Create a resource group on portal
2. Create a KV by TF with using backend=azurerm and TF ver=1.0.0 and AzureRM ver=2.71.0. Below is my config ===kv.tf===

terraform {
  backend "azurerm" {
    resource_group_name = "..."
    storage_account_name = "..."
    container_name = "..."
    key = "terraform1.tfstate"
  }
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
      version = "2.71"
    }
  }
}

provider "azurerm" {
  features {}
}

data "azurerm_client_config" "current" {}

data "azurerm_resource_group" "test" {
  name     = "..."
}

resource "azurerm_key_vault" "test" {
  name                = "..."
  location            = data.azurerm_resource_group.test.location
  resource_group_name = data.azurerm_resource_group.test.name
  sku_name = "standard"
  tenant_id = data.azurerm_client_config.current.tenant_id

  access_policy {
    tenant_id = data.azurerm_client_config.current.tenant_id
    object_id = data.azurerm_client_config.current.object_id

    key_permissions = [
      "create",
      "get",
      "list",
      "purge"
    ]

    secret_permissions = [
      "set",
      "get",
      "delete",
      "purge",
      "recover",
      "list"
    ]
  }
}

===outputs.tf===

output "key_vault_id" {
  value = azurerm_key_vault.test.id
}

4. Execute terraform apply => A KV got provisioned
5. Create a KVSec by TF. Below is my .tf config ===kvsec.tf===

data "terraform_remote_state" "test" {
  backend = "azurerm"
  config = {
    storage_account_name = "..."
    container_name       = "..."
    resource_group_name  = "..."
    key                  = "terraform1.tfstate"
  }
}

resource "azurerm_key_vault_secret" "test" {
    name  = "..."
    value = "..."
    key_vault_id = data.terraform_remote_state.test.outputs.key_vault_id
}

6. Execute terraform apply => A KVSec got provisioned underneath the KV
7. Append below section to the kvsec.tf

resource "azurerm_key_vault_secret" "test2" {
    name  = "..."
    value = "..."
    key_vault_id = data.terraform_remote_state.test.outputs.key_vault_id
}

8. Execute terraform apply => Another KVSec got provisioned underneath the KV
9. Repeat step 7 - Step 8 for say 10 times => A bunch of KVSec got provisioned underneath the KV w/o seeing tf command line errors.

@doug-papenthien-by , I also did similar things by using local state and follow your "I was using the resource ID returned by the data source to create additional azurerm_key_vault_secret". While I still could not repro.

@ all here, do you see any usage/config difference between yours and mine above?

[sinbai]

@mybayern1974 Terraform 1.0.0 azurerm 2.71.0

1. I use backend "azurerm"
2. The problem occurs in ~70% of attempts
3. KeyVault has been created separately with the local state and I don't have this issue with the local state
4. We don't use KeyVault at the same moment

@viper4u Thank you for your reply. Below are my repro steps. The issue can't be reproduced. Could you follow the stpes below to reproduce it? If not, could you provide the step-by-step to help me reproduce this issue?

My Terraform (and AzureRM Provider) Version Terraform 1.0.0 azurerm 2.71.0

Steps

1. Create a resource group named myTestResourceGroup on Azure portal.
2. Create a blob storage account named teststorageaccount on the Azure portal. (Stores the state as a Blob with the given Key within the Blob Container within the Blob Storage Account.)
3. Create a blob container named blobcontainer on the Azure portal.
4. Create a folder named ReproBug on the local machine and add a file named "step1.tf" in this folder.
5. Add the following tfconfig in step1.tf to create a azurerm_key_vault(Authenticate using a SAS Token associated with the Storage Account) :

terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
     version = "2.71.0"
    }
  }

   backend "azurerm" {
    storage_account_name = "teststorageaccount"
    container_name       = "blobcontainer"
    key                  = "prod.terraform.tfstate1"

    sas_token = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  }
}


provider "azurerm" {
  features {}
}


resource "azurerm_key_vault" "test" {
  name                       = "acctestkv-elena0901"
  location                   = "westus2"
  resource_group_name        = "myTestResourceGroup"
  tenant_id                  = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  sku_name                   = "standard"
  soft_delete_retention_days = 7

  access_policy {
    tenant_id = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
    object_id = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

    key_permissions = [
      "Get",
	  "Delete",
    ]

    secret_permissions = [
      "Get",
      "Delete",
      "List",
      "Purge",
      "Recover",
      "Set",
    ]
  }

  tags = {
    environment = "Production"
  }
}

output "key_vault_id" {
  value = azurerm_key_vault.test.id
}

6. Run terraform init、terraform plan、 terraform apply to create azurerm_key_vault.
7. Add a file named "step2.tf" in ReproBug folder.
8. Add the following tfconfig in step2.tf to create a azurerm_key_vault_secret(Authenticate using a SAS Token associated with the Storage Account):

terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
     version = "2.71.0"
    }
  }

   backend "azurerm" {
    storage_account_name = "teststorageaccount"
    container_name       = "blobcontainer"
    key                  = "prod.terraform.tfstate2"

    sas_token = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  }
}

provider "azurerm" {
  features {}
}

data "terraform_remote_state" "remotestate" {
  backend = "azurerm"

  config = {
    storage_account_name = "teststorageaccount"
    container_name       = "blobcontainer"
    key                  = "prod.terraform.tfstate1"

    sas_token = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  }
}

resource "azurerm_key_vault_secret" "mysecretvalue" {
  name         = "MySecretValue-0901"
  value        = "test"
  key_vault_id = data.terraform_remote_state.remotestate.outputs.key_vault_id
}

9. After successfully creating azurerm_key_vault, run terraform init、terraform plan、 terraform apply to create azurerm_key_vault_secret=>azurerm_key_vault_secret was created,no command line error msg

10. Continue to add ten azurerm_key_vault_secrets with different key and different azurerm_key_vault_secret name to the azurerm_key_vault which created in step 6.=>All of them worked well, There is no command line error msg in my test

[MaksymChornyi]

@sinbai @mybayern1974 Sorry for the late answer I tried to reproduce it with a simple script but I haven't faced the problem. I have a failed attempt with a trace log level. Where can I send it?

[sinbai]

@sinbai @mybayern1974 Sorry for the late answer I tried to reproduce it with a simple script but I haven't faced the problem. I have a failed attempt with a trace log level. Where can I send it?

@viper4u Mail log to [email protected], thank you. I will try to see if there are any findings in the log, but I think step-by-step-repro-this-issue is the key to investigating this issue. Thank you very much.

[MaksymChornyi]

@sinbai I've reproduced it with a simple script. I have an Azure KeyVault and that issue is reproduced each time with that one How can I help you to with it?

[sinbai]

@sinbai I've reproduced it with a simple script. I have an Azure KeyVault and that issue is reproduced each time with that one How can I help you to with it?

Can I access that Azure KeyVault? And could you share your script (contains the access token) with me to reproduce it on my side?

[MaksymChornyi]

What did you mean by "access token"? I use Service Principal auth

[MaksymChornyi]

I've sent all details to your mail

[sinbai]

I've sent all details to your mail

Thank you. Per details provided, I cannot access the Azure KeyVault due to the access permissions.

[MaksymChornyi]

I've reproduced the issue with new keyvault Context:

• Windows 10
• Terraform version: 1.0.6
• Service principle auth

Step to reproduce:

1. I am from East Europe
2. Create a new KeyVault in East US
3. Run simplest terraform script

terraform {

  backend "local" {
  }

  required_providers {
   
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 2.75.0"
    }
   
  }
}
provider "azurerm" {
  features {}
}

resource "azurerm_key_vault_secret" "test_key" {
  name         = "test"
  value        = "test"
  key_vault_id = "/subscriptions/<subscription_id>/resourceGroups/<rg-name>/providers/Microsoft.KeyVault/vaults/<vault_name>"
}

4. Secret has been created
5. Terraform output is "Error: Provider produced inconsistent result after apply"

[sinbai]

@viper4u Per the log provided, I found the Azure keyVault cannot be found after setting the secret, Could you help confirm whether the keyvault exists after reproducing this issue?

It is best to confirm through the following link： https://docs.microsoft.com/en-us/rest/api/resources/resources/list#code-try-0

Notes:

1. Update the API version with 2020-06-01.
2. Add the optional parameter $filter with value resourceType eq 'Microsoft.KeyVault/vaults' and name eq '<kv name>' like this:
3. Try it several times to see if you can find it every time.

In addition, could you also check the total resources count of your subscription?

[MaksymChornyi]

Results

1. when I use the filter with "and name eq ''", then I get { "value": [], "nextLink": "<some_url>" }
2. when I use the filter without "and name eq ''", then I get list of resources with my KeyVault
3. When I try another Keyvault in another subscription then I get result without nextlink { "value": [ { "id": "/subscriptions/<my_subscription>/resourceGroups/<rg_name>/providers/Microsoft.KeyVault/vaults/<kv_name>", "name": "makctestkv3", "type": "Microsoft.KeyVault/vaults", "location": "eastus2" } ] }

[sinbai]

Results

1. when I use the filter with "and name eq ''", then I get { "value": [], "nextLink": "<some_url>" }
2. when I use the filter without "and name eq ''", then I get list of resources with my KeyVault
3. When I try another Keyvault in another subscription then I get result without nextlink { "value": [ { "id": "/subscriptions/<my_subscription>/resourceGroups/<rg_name>/providers/Microsoft.KeyVault/vaults/<kv_name>", "name": "makctestkv3", "type": "Microsoft.KeyVault/vaults", "location": "eastus2" } ] }

Could you please share with me the screenshot of the above results and obscure sensitive information(Including parameter settings and respond body)?

[sinbai]

I submitted a PR that might solve this issue. Since I can’t reproduce this problem until now, I can’t verify it. Is anyone willing to build it and verify that this issue is fixed? The info to build a TF provider (on Windows) are as follows:

Requirements: Terraform version 0.12.x + (but 1.x is recommended) Go version 1.16.x (to build the provider plugin) Git Bash for Windows Make for Windows

Using the locally compiled Azure Provider binary: For example, add the following to terraform.rc for a provider binary located in D:\GoCode\bin. The file named terraform.rc and placed in the relevant user's %APPDATA% directory.

Steps:

1. Pull this PR
2. Run make build in git bash.

After make build is completed, it will be used automatically when running terraform apply.

References: https://github.com/hashicorp/terraform-provider-azurerm/blob/main/README.md

[MaksymChornyi]

@sinbai We have tried your PR #13409 and it solves our problem

[tplive]

@sinbai I'm trying to confirm if this solved the problem, so I've created a mock tf setup, and I'm deploying to the same subscription using the same SP as where I had the problem, but running only the creation of the vault secret that fails in production. I'm getting the same error:

azurerm_key_vault_secret.domain-join-password: Creating...

│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to azurerm_key_vault_secret.domain-join-password, provider "provider[\"registry.terraform.io/hashicorp/azurerm\"]" produced
│ an unexpected new value: Root resource was present, but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

However, I'm new to running developer versions of the provider, so I'm unsure if I did it correctly. It's still downloading the azurerm 2.78 version, even though I'm warned that I'm using developer override. Is it supposed to still dl the prod version? 🤷🏽

[sinbai]

@sinbai We have tried your PR #13409 and it solves our problem

Thank you for your kind cooperation.

[sinbai]

However, I'm new to running developer versions of the provider, so I'm unsure if I did it correctly. It's still downloading the azurerm 2.78 version, even though I'm warned that I'm using developer override. Is it supposed to still dl the prod version? 🤷🏽

@tplive It will be downloaded if you run terraform init when using provider development overrides. if so, please skip terraform init, It is not necessary and may error unexpectedly.

BTW, The PR #13409 depends on the trace log and confirmation information provided by viper4u. The same error maybe caused by different reasons, so, if the tf config and trace log (after applying PR #13409) can be provided, I would try to find the reason for the failure.

[tplive]

@sinbai thank you for the advice. I've stripped it down further, so the azurerm provider is the only thing remaining, it deploys only a single kv secret now, and still fails.. :) I can send you the config and trace log if you like to see what that looks like?

[sinbai]

@tplive Please send the config and trace log to [email protected], thank you.

[dma-sitecore]

@tombuildsstuff @sinbai The response from Azure Support

1. The primary recommendation to resolve this issue is to move to Azure Resource Graph instead of the List Resources API. You can ignore all of the API implementation details and nextLink challenges and just make a single call to Search-AzGraph. When the portal displays a list of resources, it is done using Azure Resource Graph rather than the List Resources API. You can replace all of your code with just this one line:

Search-AzGraph -Query "resources | where type =~ 'Microsoft.KeyVault/vaults' and name =~ '$AzureKeyVaultName'" -Subscription $SubscriptionId

[tplive]

@tombuildsstuff I have been working with @sinbai on reproducing the issue and testing the #13409 patch, and I was able to accurately repro the bug in our production environment, both with the latest 2.79.0 version and confirmed that the patched version also still has the bug in our case. However, we are not allowed to send tracelogs from our production environment to external parties, so we are unable to participate in further testing, since our test environment does not repro the bug, unfortunately. I will still closely monitor this issue, as it is still important to us to have it resolved. Thanks for all your great help and patience @sinbai !!