GCP IAP appears to be incompatible with the terraform provider

[apex-omontgomery]

tl;dr: If you setup Identity Aware Proxy to secure endpoint access, when accessing the endpoint from the provider returned HTML causes the provider to choke on the response. This is probably an unsupported feature, but outside of split DNS which creates a secured endpoint, and an insecure endpoint I can't think of a way to interact with the vault instance.

Terraform Version

$ terraform -v
Terraform v0.14.7
+ provider registry.terraform.io/hashicorp/google v3.40.0
+ provider registry.terraform.io/hashicorp/google-beta v3.40.0
+ provider registry.terraform.io/hashicorp/vault v2.19.0

Your version of Terraform is out of date! The latest version
is 0.15.0. You can update by downloading from https://www.terraform.io/downloads.html

Affected Resource(s)

Please list the resources as a list, for example:

• provider "vault"

This is related with the core systems that handle interaction with the REST api.

Terraform Configuration Files

This requires a working vault instance on GCP that is using IAP.

provider "vault" {
  address = "https://tst-vault.corp.company.com:443"
  token = "<ROOT_TOKEN>"
  token_name = "terraform_policy_token"
}

I can replicate this issue also by using the vault CLI, but I can get around this by interacting directly with the rest API and using these steps

$ vault read auth/sys
Error reading auth/sys: invalid character '<' looking for beginning of value

Debug Output

I can provider a gist if needed. I would need to properly filter out any portions that can be considered sensitive. For brevity here are the last few lines:

$ TF_LOG=DEBUG terraform plan
....
2021/04/26 09:47:20 [WARN] Provider "registry.terraform.io/hashicorp/google" produced an unexpected new value for google_project_iam_member.example during refresh.
      - .etag: was cty.StringVal("BwW/lHTkEuw="), but now cty.StringVal("BwXAHhI1eeI=")
2021-04-26T09:47:20.673-0500 [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google/3.40.0/linux_amd64/terraform-provider-google_v3.40.0_x5 pid=28523
2021-04-26T09:47:20.673-0500 [DEBUG] plugin: plugin exited
2021/04/26 09:47:20 [INFO] backend/local: plan operation completed

Error: invalid character '<' looking for beginning of value

  on provider.tf line 6, in provider "vault":
   6: provider "vault" {


2021-04-26T09:47:20.677-0500 [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/vault/2.19.0/linux_amd64/terraform-provider-vault_v2.19.0_x4 pid=28504
2021-04-26T09:47:20.677-0500 [DEBUG] plugin: plugin exited

Panic Output

N/A

Expected Behavior

The failure that is currently happening. I don't expect the provider to know how to handle IAP without some sort of configuration. The problem is how to configure this.

I'd like the ability to either work with IAP, or a recommended course of action to handle these situations.

Actual Behavior

Same as above.

Steps to Reproduce

1. Create a vault instance.
2. Configure IAP to proxy the vault instance.
3. Try to interact using the vault terraform provider.

Important Factoids

The only thing I've found to get around this issue is to have separate endpoints for IAP and non-IAP. And since vault expects a certain DNS name for the api_addr the only way I've found is to use split DNS. But all this does is provide a mechanism to bypass IAP.

References

https://cloud.google.com/iap/docs/authentication-howto https://cloud.google.com/iap/docs/enabling-kubernetes-howto https://cloud.google.com/iap

This tutorial looks similar to what I did- https://blog.doit-intl.com/vault-high-availability-on-gke-68ef4fd7ca33

[bogdan-fusa-visma]

I've run into the same issue. Was this fixed or is there another way of getting vault that is behind IAP work with the provider?

terraform/atlantis plan is returning `Error: Error making API request.

URL: GET https://accounts.google.com/o/oauth2/v2/auth Code: 404. Raw Message:`

[rwblokzijl]

We got pretty far with it:

data "google_service_account_id_token" "vault_iap_oidc" {
  # TODO: get from env
  target_service_account = var.vault_service_account
  target_audience        = var.vault_iap_client_id
  include_email          = true
}

provider "vault" {
  address = var.vault_address
  auth_login_jwt {
    mount = "gcp"
    role  = var.vault_role
    jwt   = null # Gotten from env TERRAFORM_VAULT_AUTH_JWT
  }
  headers {
    name  = "Proxy-Authorization"
    value = "Bearer ${data.google_service_account_id_token.vault_iap_oidc.id_token}" # TODO: get OIDC from env
  }
}

However if there is too much time between Plan and Apply it fails. This is because the token is generated at plan time, and might have expired when the plan is applied. This is not an issue if you run terraform from your terminal, but we have a workflow where we create a PR for the terraform code. A CICD generates a tfplan file, which we then review internally. If all looks good, we then merge the PR which triggers the CICD to apply the generated plan file. If our review takes longer than the expiry time of the token, then the provider will fail.

If this provider had an environment variable that allowed us to add custom headers that would allow us to fix the issue. (https://github.com/hashicorp/terraform-provider-vault/issues/1734)