Intermittent Error: Unable to locate Storage Account when RAGRS/RAZGRS account kind is used

[andrey-dubnik]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

2.29.0

Affected Resource(s)

azurerm_storage_account intermittently produces an error on plan for the already created resource Error: Unable to locate Storage Account

Terraform Configuration Files

resource "azurerm_resource_group" "example" {
  name     = "example-resources"
  location = "West Europe"
}

resource "azurerm_storage_account" "example" {
  name                     = "storageaccountname"
  resource_group_name      = azurerm_resource_group.example.name
  location                 = azurerm_resource_group.example.location
  account_tier             = "Standard"
  account_replication_type = "RAZGRS"
}

Expected Behaviour

Should be no error for the RA and non RA accounts

Actual Behaviour

intermittently produces an error on plan for the already created resource Error: Unable to locate Storage Account

Steps to Reproduce

1. terraform apply
2. terraform plan

Important Factoids

Only affecting RA storage accounts

[magodo]

@andrey-moor I've applied the configuration as you provided above (a typo in the account_replication_type, which should be RAGZRS rather than RAZGRS). Then I started a loop to run terraform plan until it returns error. With a bunch of iterations, the issue doesn't appear. So would you please follow the terraform debug guide and provide the debug log here? Mostly that is due to some API issues, while we will need the "request id" and other context info for further contacting the Azure supports.

[andrey-dubnik]

Forgot to mention the issue was reproducible only via GH actions and was intermittent. It may be down to the Azure portal replication etc. as GH hosts runners in multiple location and 1st available runner is selected for the pipeline.

East US (eastus) East US 2 (eastus2) West US 2 (westus2) Central US (centralus) South Central US (southcentralus)

Similar behaviour is affecting the KeyVault but with much less frequency

Let me see if I can reproduce the issue and capture the debug data via the standalone pipeline, we have switched to ZGRS accounts to unblock the delivery so original config is not available.

[varnav]

I'm seeing this on creation:

Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="StorageAccountNotFound" Message="The storage account testsa was not found."

with 2.93.1 with ZGRS type.

Related: https://github.com/hashicorp/terraform-provider-azurerm/issues/5299#issuecomment-980079517

[andrey-dubnik]

@magodo sorry took me a while, I have captured the trace debug of the problem.

[andrey-dubnik]

suspect it is related to the call which comes back empty - https://docs.microsoft.com/en-us/rest/api/storagerp/storage-accounts/list

2022-02-03T16:14:30.610Z [DEBUG] provider.terraform-provider-azurerm_v2.94.0_x5: AzureRM Response for https://management.azure.com/subscriptions/264f194d-71cc-41d9-9bf4-c4f5456285e8/providers/Microsoft.Storage/storageAccounts?api-version=2021-04-01: 
HTTP/2.0 200 OK
Cache-Control: no-cache
Content-Type: application/json; charset=utf-8
Date: Thu, 03 Feb 2022 16:14:30 GMT
Expires: -1
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: fc3a2733-bb7a-9fdc-09e6-d9510d90bf10
X-Ms-Request-Id: 612cc49f-6921-4522-9a25-c7eaa4e7b3b2
X-Ms-Routing-Request-Id: EASTUS2:20220203T161430Z:612cc49f-6921-4522-9a25-c7eaa4e7b3b2

{"value":[]}: timestamp=2022-02-03T16:14:30.609Z

[andrey-dubnik]

Seem the workaround is to create a TAG on the storage account (via portal), after that the issue goes away and above API returns the value. More to it it started to return value for all the storage account in that sub which were having the issue...

[andrey-dubnik]

The workaround came from the #11059 as we've seen similar but not that permanent issue with the keyvault. The root cause is highly likely to be the same for both of those issues.

[magodo]

@andrey-dubnik Just to be sure, the LIST call for the SA on your test sub always return an empty list, even after waiting for a long while? Is there any other SA in that sub?

[andrey-dubnik]

This is correct, terraform was able to obtain the keys and all the data for the account but the list api returned blanc hence the account can't be found error.

There was another account in the sub which was having an issue originally so in total there are 2 accounts in there. After tagging at least one the second account also appeared in the api call.

[mariussm]

I can add that tagging a storage account worked on my side as well. My storage account was created like this:

resource "azurerm_storage_account" "sa" {
    name                     = var.storage_account_name
    location                 = var.location
    account_tier             = "Standard"
    account_replication_type = "GRS"
    resource_group_name      = azurerm_resource_group.rg.name
}

///It worked 1 out of 8 times, or something, before adding a manual tag. Now it seems stable. ///

Edit: Nope, it is not stable at all with the tag thing either.

[roehrijn]

This issue is also causing sleepless nights on our side here. What I can say so far: most likely this issue is caused by some race condition hitting sorts of ARM API limits. We're experiencing this on a Terraform project with around 100 resources (1 storage account, 1 key vault, a lot of role assignments to them) and I'm not able to reproduce the issue in a smaller project. However, calling terraform apply -parallelism=1 seems to help.

Furthermore: We're experiencing the very same behaviour also on Key Vaults (is switches randomly from Storage Account to Key Vault, whatever Terraform tries to access first). Here is a log example from that:

    2022-03-03T06:57:41.0852887Z 2022-03-03T06:57:40.565Z [INFO]  Starting apply for azurerm_key_vault_secret.githubtoken
    2022-03-03T06:57:41.0853464Z 2022-03-03T06:57:40.565Z [DEBUG] azurerm_key_vault_secret.githubtoken: applying the planned Update change
    2022-03-03T06:57:41.0854255Z 2022-03-03T06:57:40.567Z [INFO]  provider.terraform-provider-azurerm_v2.98.0_x5: preparing arguments for AzureRM KeyVault Secret update.: timestamp=2022-03-03T06:57:40.566Z
    2022-03-03T06:57:41.0854946Z 2022-03-03T06:57:40.567Z [DEBUG] provider.terraform-provider-azurerm_v2.98.0_x5: AzureRM Request: 
    2022-03-03T06:57:41.0855810Z GET /subscriptions/***/resources?%24filter=resourceType+eq+%27Microsoft.KeyVault%2Fvaults%27+and+name+eq+%27net-prod-839977%27&%24top=5&api-version=2020-06-01 HTTP/1.1
    2022-03-03T06:57:41.0856269Z Host: management.azure.com
    2022-03-03T06:57:41.0857164Z User-Agent: Go/go1.17.5 (amd64-linux) go-autorest/v14.2.1 Azure-SDK-For-Go/v61.4.0 resources/2020-06-01 HashiCorp Terraform/1.1.6 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/2.98.0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
    2022-03-03T06:57:41.0857908Z X-Ms-Correlation-Request-Id: b01b3074-24a4-9613-dd3f-c0a93338159e
    2022-03-03T06:57:41.0858375Z Accept-Encoding: gzip: timestamp=2022-03-03T06:57:40.566Z
    2022-03-03T06:57:41.0859118Z 2022-03-03T06:57:40.635Z [ERROR] vertex "azurerm_key_vault_secret.githubtoken" error: Unable to determine the Resource ID for the Key Vault at URL "https://net-prod-839977.vault.azure.net/"
    2022-03-03T06:57:41.0860401Z 2022-03-03T06:57:40.635Z [DEBUG] provider.terraform-provider-azurerm_v2.98.0_x5: AzureRM Response for https://management.azure.com/subscriptions/***/resources?%24filter=resourceType+eq+%27Microsoft.KeyVault%2Fvaults%27+and+name+eq+%27net-prod-839977%27&%24top=5&api-version=2020-06-01: 
    2022-03-03T06:57:41.0861012Z HTTP/2.0 200 OK
    2022-03-03T06:57:41.0861306Z Cache-Control: no-cache
    2022-03-03T06:57:41.0861671Z Content-Type: application/json; charset=utf-8
    2022-03-03T06:57:41.0861971Z Date: Thu, 03 Mar 2022 06:57:39 GMT
    2022-03-03T06:57:41.0862330Z Expires: -1
    2022-03-03T06:57:41.0862605Z Pragma: no-cache
    2022-03-03T06:57:41.0863019Z Strict-Transport-Security: max-age=31536000; includeSubDomains
    2022-03-03T06:57:41.0863386Z Vary: Accept-Encoding
    2022-03-03T06:57:41.0863725Z X-Content-Type-Options: nosniff
    2022-03-03T06:57:41.0864181Z X-Ms-Correlation-Request-Id: b01b3074-24a4-9613-dd3f-c0a93338159e
    2022-03-03T06:57:41.0864686Z X-Ms-Ratelimit-Remaining-Subscription-Reads: 11997
    2022-03-03T06:57:41.0865152Z X-Ms-Request-Id: f6f7065a-d272-43ec-9bf3-526a820189ac
    2022-03-03T06:57:41.0865654Z X-Ms-Routing-Request-Id: WESTUS3:20220303T065740Z:f6f7065a-d272-43ec-9bf3-526a820189ac
    2022-03-03T06:57:41.0865936Z 
    2022-03-03T06:57:41.0866144Z {"value":[]}: timestamp=2022-03-03T06:57:40.634Z

[mariussm]

@roehrijn , I have tried with parallelism 1 now, and it does not resolve any issue when it comes to storage accounts at least. Maybe that workaround only works for key vaults?

[roehrijn]

Hi @mariussm, it also works for storage accounts in my environment. However, as a wrote, this is unfortunately likely to be some sort of race condition in rate limiting. That's why I think parallelism=1 is not a 100% fix/workaround. Hope MS is going to address this soon.

[damoodamoo]

@roehrijn / @andrey-dubnik - we've experienced this issue over the past month or so: We found that a call to the /resources?%24filter=resourceType+eq+%27Microsoft.KeyVault% endpoint provided different results depending on the region that serviced that ARM API call. Essentially if the X-Ms-Routing-Request-Id value was the same as the location of the resource, the call returned the key vault - but if not it sporadically would return an empty array. This had a downstream effect on the rest of the TF deployment as it treated the resource as missing. It would be interesting for you to check the X-Ms-Routing-Request-Id of your calls to see if there's a correlation between successful and failed ones.

We have an open case with the ARM API team, and so far they've confirmed that it's an issue with the ARM API cross-region Cache not being updated quick enough.

In terms of fixes / workarounds - they're currently a bit limited:

• Recommendation from the API team is to use the Azure Resource Graph for list operations, which has a 1 minute SLO for replication. Potentially a pretty big change to the provider code.
• Wait for the replication issues to die down + backend to improve
• Find some way of manually targeting the ARM API at the location that contains the resource

cc: @stuartleeks

[andrey-dubnik]

Using tags as a workaround worked so far and the portal cache was replicated. There were no re-occurrences of the issue since tagging.

Since this is a provider-api scope there is no way to influence it externally. If this is a replication lag and not a permanent issue then adding a retry logic would probably help in mitigating the issue as worst case it would be 1 min SLO in oppose to an error which is already good enough.

If tagging permanently fixes the issue maybe the api team can use this in the replication fix...

[292368]

Hello .

I had also experienced same Intermittent issue i.e Unable to locate Storage Account for GRS S account. Root cause of this issue is ARM cache sync issue. Contact to Microsoft to resync the storage account .

It working for me post resync of ARM cache

Thanks Tushar P

[paweltucholskiigt]

I got this issue also. az storage account list was listing [] adding tag did fix the problem so indeed this looks like stale cache problem.

[imduchy]

We're experiencing this problem in GitHub actions too.

Run hashicorp/[email protected]

provider registry.terraform.io/hashicorp/azurerm v3.10.0

[AvtsVivek]

I am experiencing the same problem on my local machine. Here is the repo.

I run these commands and I encounter this specifically when I run the destroy commands.

terraform plan -destroy -out main.destroy.tfplan

terraform apply main.destroy.tfplan

│ Error: Unable to locate Storage Account "staticwebsiteprfpim"! │

The following shows plan command.

I get that error when I run the command for the first time. When I re-run the same command the second time, things run fine.

Same is the case with apply command as well.

[paalders]

When using GH Actions in combination with AzureRM and account_replication_type LRS you have the same problem. When I run the same terraform apply locally I don't have any issues. Seems related to GH actions.

azrumrm version 3.12.0

resource "azurerm_storage_account" "redacted" {
  name                     = local.storage_account_name
  resource_group_name      = azurerm_resource_group.redacted.name
  location                 = azurerm_resource_group.redacted.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

[ekristen]

Using azurerm 3.16.0, storage account type LRS, this happens fairly frequently.

[magodo]

@ekristen and @paalders could you please help confirm that the storage account list API (i.e. /subscriptions/{subscriptionId}/providers/Microsoft.Storage/storageAccounts) doesn't return the expected storage account, while the ARG API does? If that is the case, we might consider migrating to use ARG instead.

[AndrzejK-Atende]

Same problem in my case:

Terraform v.1.2.6 azurerm: v.3.17.0

It does not count if SA is LRS, od ZRS, 7 times on 10 I get error: "Unable to locate Storage Account".

[MLKiiwy]

Also same issue for me on github actions + terraform cloud. Also LRS type for replication. The storage account cannot be found.

I can see it on the azure dashboard and on my local machine the command line tool return the correct list.

Terraform v.1.2.6 azurerm: v.3.17.0

[magodo]

@MLKiiwy Just want to be sure, which command you were running that returns you the correct list? Is it the "resource list" or Azure Resource Graph query?

[as9333]

I am facing the same issue. with azurerm 3.21.0. Any workarounds ?

[avooca]

Randomly impacting LRS Storage accounts azurerm 3.5.0. / TFC Quite annoying...

[ghost]

Got the same issue, just created a storage account through TF and 5 minutes later it can't find it.

[b-c-lucas]

Still seeing this for LRS storage accounts as of Terraform CLI 1.3.7 + azurerm 3.37.0.

Seems like it's been an issue for years at this point: #5299

[ekristen]

This is still a problem. Right now I have a terraform that simply won't find a storage account on refresh even though it's clearly right there.