Vault provider doesn't wait for variables in configuration

[svjakrm]

I have next vault provider configuration:

provider "vault" {
  address = "http://${data.kubernetes_service.vault.status[0].load_balancer[0].ingress[0].ip}:8200"
  token   = data.google_kms_secret.keys.plaintext
}
data "kubernetes_service" "vault" {
  metadata {
    name      = "vault"
    namespace = module.vault.namespace
  }
 depends_on = [module.vault]
}
data "google_kms_secret" "keys" {
  crypto_key = module.vault.kms_crypto_key
  ciphertext = data.http.keys.body
  depends_on = [module.vault]
}

And even with depends_on in every data resource, vault provider doesn't wait until dependant modules become ready and provide data and fail on plan execution with next error

│ Error: no vault token found
│ 
│   with provider["registry.terraform.io/hashicorp/vault"],
│   on vault.tf line 12, in provider "vault":
│   12: provider "vault" {
│ 

I have another providers in the same configuration, which successfully works with variables like

provider "helm" {
  kubernetes {
    host                   = module.gke.endpoint
    cluster_ca_certificate = base64decode(module.gke.cluster_ca_certificate)
    token                  = data.google_client_config.current.access_token
  }
}

provider "kubernetes" {
  host = module.gke.endpoint

  cluster_ca_certificate = base64decode(module.gke.cluster_ca_certificate)
  token                  = data.google_client_config.current.access_token
}

They works fine and wait, until module.gke data become available. Can we fix vault provider?

[iiro]

Thanks for raising this @svjakrm - bumped into the same problem also. And, no proper / real workarounds on this one... :/

[seanamos]

What's strange about this is I have a linux machine where this works fine. On my macbook it doesn't work, same issue.

[seanamos]

After looking at the provider source code a bit, I've figured out what's going on here and why it works on my one machine but not my other machine.

The provider eagerly validates that the token isn't empty here: https://github.com/hashicorp/terraform-provider-vault/blob/7e961d6739d9d459d15ce8a1628fa92cd1b8a2e6/internal/provider/meta.go#L210 This is where the provider errors out too early if your trying to use a data source to fetch the token.

However, currently it doesn't need to actually use the token that it validates! If you've logged into vault before, you will have a ~/.vault-token file and it will read it from there, or if you have the env var VAULT_TOKEN set it will read it from there. These don't actually need to be valid tokens, they can be set to anything and the validation will pass. eg. export VAULT_TOKEN=dummy

Once the validation has passed, the provider will actually read the token from specified source in TF, ie. token = data.google_kms_secret.keys.plaintext.

On my Linux machine where I thought this worked, I had logged into vault before, so a ~/.vault-token file existed (containing a long expired token) which let the validation pass.

In short, the workaround:

export VAULT_TOKEN=not_a_real_token
terraform apply # no error

This is some pretty strange behavior and doesn't line up with how the other providers work (consul/nomad), so definitely a bug.

[krarey]

An additional data point: if using namespaces in Vault Enterprise, you additionally have to set skip_child_token = true to employ this workaround, as the Vault provider attempts to lookup the namespace of the temporary VAULT_TOKEN during initialization.

[tmatilai]

I think this is related to (at least) #192 and #1541. The issue is not limited to the token, but also e.g. dynamic address attribute verification fails in some cases.

https://github.com/hashicorp/terraform-provider-vault/pull/1541#issuecomment-1291032715 gives hope that it will be addressed at some point. Hopefully soon! 🤗

We currently are tracking this feature request in our internal backlog, so it will make it into an upcoming release. We will definitely link any related work to this PR.

[tmatilai]

We currently are tracking this feature request in our internal backlog, so it will make it into an upcoming release. We will definitely link any related work to this PR.

@benashz sorry for the ping, but are there any plans if/when this could get into the roadmap?

Another use case would be to fetch the admin token from HCP when initially configuring other Vault authentication methods.

[livioribeiro]

I am trying to setup a nomad cluster with vault using Terraform, and also stumbled upon this behavior on the provider.

To work around this, I am calling another Terraform project using the null provider, saving the outputs to a file and reading them back on the "parent" project.

It's ugly, but it works.

[iniinikoski]

We currently are tracking this feature request in our internal backlog, so it will make it into an upcoming release. We will definitely link any related work to this PR.

@benashz sorry for the ping, but are there any plans if/when this could get into the roadmap?

Another use case would be to fetch the admin token from HCP when initially configuring other Vault authentication methods.

Looking also forward for this - use cases are also initial bootstrapping and a complete CI between HCP and Vault provider within a single run - no trickeries needed.

[tmatilai]

If I get it right, this is being addressed in #2049

[fairclothjm]

The fix is available in https://github.com/hashicorp/terraform-provider-vault/releases/tag/v3.23.0