terraform-provider-vault icon indicating copy to clipboard operation
terraform-provider-vault copied to clipboard

Bug - vault_kubernetes_auth_backend_config - vault enterprise - required index state not present

Open drtbz opened this issue 2 years ago • 2 comments

Terraform Version

Terraform v1.1.6 on darwin_amd64

Vault provider 3.3.1

Affected Resource(s)

Please list the resources as a list, for example:

  • vault_kubernetes_auth_backend_config
  • probably all resources that read-after-write

Terraform Configuration Files

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key.

resource "vault_kubernetes_auth_backend_config" "kube_config" {
  backend                 =  var.vault_auth_path
  kubernetes_host         =  var.kubernetes_hostname
  kubernetes_ca_cert      =  var.kubernetes_ca_cert
  token_reviewer_jwt      =  lookup(data.kubernetes_secret.vault_token,"token", "")
  issuer                  =  "api"
  disable_iss_validation  =  "true"
  depends_on              = [kubernetes_cluster_role_binding.role_tokenview_binding]

}

Expected Behavior

vault_kubernetes_auth_backend_config resource is created and no errors

Actual Behavior

vault_kubernetes_auth_backend_config resource is created and intermittently throws:

Code: 412. Errors:
        	            	│ 
        	            	│ * required index state not present

Important Factoids

I'm pretty sure we're running into this read-after-write issue - as we have a multi-cloud vault enterprise solution.

This didn't seem to happen on previous versions of the vault provider - but we did a relatively large upgrade from v2.24.1

Hopefully it's as easy a fix in the provider as looking for the right headers, or simply introducing a short back-off between resource creation and reading back it's configuration.

drtbz avatar Mar 18 '22 11:03 drtbz

probably all resources that read-after-write <--- seeing this behavior for the vault_database_secret_backend_role module as well

mough avatar Jun 01 '22 14:06 mough

So I'm pretty sure it's related to https://github.com/hashicorp/terraform-provider-vault/blob/83b888304a2a0b17779a14364d2d401836a97b97/vault/resource_kubernetes_auth_backend_config.go#L144

The return function kubernetesAuthBackendConfigRead is immediately trying to read back the config data and doesn't seem to respect retrying on a 412

drtbz avatar Aug 03 '22 11:08 drtbz

This is happening when the Terraform provider authenticates to Vault as well. It creates an ephemeral token and then tries to do a token lookup, and it also hits 412. The result is that the apply operation fails and has to be retried. Errors that we are seeing:

Error: Error making API request.

URL: GET https://urlredacted.com/v1/auth/token/lookup-self
Code: 412. Errors:

* required index state not present

  with module.postgresdb.provider["registry.terraform.io/hashicorp/vault"].local,
  on postgresdb/providers.tf line 14, in provider "vault":
  14: provider "vault" {

AdrienneCohea avatar Jul 28 '23 00:07 AdrienneCohea