[Bug]: Provider crashes when you want to read multiple resources, but OIDC fails to provide a token before timing out

[danielskowronski]

Terraform Core Version

1.7.0

Terraform Vault Provider Version

3.24.0

Vault Server Version

1.13.3

Affected Resource(s)

At least:

• vault_kv_secret_v2

Expected Behavior

When using OIDC to login and reading more than one vault_kv_secret_v2, the provider should gracefully time out on waiting to obtain a token.

Actual Behavior

It crashes in a nasty way.

Relevant Error/Panic Output Snippet

data.vault_kv_secret_v2.some_secret: Reading...
data.vault_kv_secret_v2.some_other_secret: Reading...
data.vault_kv_secret_v2.some_other_secret: Still reading... [10s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [10s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [20s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [20s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [30s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [30s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [40s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [40s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [50s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [50s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [1m0s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [1m0s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [1m10s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [1m10s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [1m20s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [1m20s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [1m30s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [1m30s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [1m40s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [1m40s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [1m50s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [1m50s elapsed]
data.vault_kv_secret_v2.some_other_secret: Still reading... [2m0s elapsed]
data.vault_kv_secret_v2.some_secret: Still reading... [2m0s elapsed]
╷
│ Error: Request cancelled
│
│   with data.vault_kv_secret_v2.some_secret,
│   on test.tf line 19, in data "vault_kv_secret_v2" "some_secret":
│   19: data "vault_kv_secret_v2" "some_secret" {
│
│ The plugin.(*GRPCProvider).ReadDataSource request was cancelled.
╵
╷
│ Error: Timed out waiting for response from provider
│
│   with data.vault_kv_secret_v2.some_other_secret,
│   on test.tf line 23, in data "vault_kv_secret_v2" "some_other_secret":
│   23: data "vault_kv_secret_v2" "some_other_secret" {
│
╵

Stack trace from the terraform-provider-vault_v3.24.0_x5 plugin:

panic: http: multiple registrations for /oidc/callback

goroutine 39 [running]:
net/http.(*ServeMux).Handle(0x104c82c40, {0x10393e25d, 0xe}, {0x1041b1220?, 0x14000a89540})
    net/http/server.go:2530 +0x1f8
net/http.(*ServeMux).HandleFunc(...)
    net/http/server.go:2567
net/http.HandleFunc(...)
    net/http/server.go:2579
github.com/hashicorp/vault-plugin-auth-jwt.(*CLIHandler).Auth(0x14000d0a750?, 0x1400074a500, 0x14000d0aea0)
    github.com/hashicorp/[email protected]/cli.go:125 +0x430
github.com/hashicorp/terraform-provider-vault/internal/provider.(*AuthLoginOIDC).Login(0x14000d0a750?, 0x103936d6f?)
    github.com/hashicorp/terraform-provider-vault/internal/provider/auth_oidc.go:116 +0x6c
github.com/hashicorp/terraform-provider-vault/internal/provider.(*ProviderMeta).setClient(0x14000a507c0)
    github.com/hashicorp/terraform-provider-vault/internal/provider/meta.go:292 +0xb38
github.com/hashicorp/terraform-provider-vault/internal/provider.(*ProviderMeta).getClient(...)
    github.com/hashicorp/terraform-provider-vault/internal/provider/meta.go:403
github.com/hashicorp/terraform-provider-vault/internal/provider.(*ProviderMeta).GetClient(0x14000a507c0)
    github.com/hashicorp/terraform-provider-vault/internal/provider/meta.go:64 +0x70
github.com/hashicorp/terraform-provider-vault/internal/provider.GetClient({0x104177760?, 0x14000c30100?}, {0x104101c80?, 0x14000a507c0})
    github.com/hashicorp/terraform-provider-vault/internal/provider/meta.go:488 +0x2ac
github.com/hashicorp/terraform-provider-vault/vault.kvSecretV2DataSourceRead({0x1041c0f58?, 0x14000c04450?}, 0x104c83500?, {0x104101c80?, 0x14000a507c0?})
    github.com/hashicorp/terraform-provider-vault/vault/data_source_kv_secret_v2.go:94 +0x44
github.com/hashicorp/terraform-provider-vault/vault.kvSecretV2DataSource.ReadContextWrapper.func1({0x1041c1000, 0x14000b251f0}, 0x0?, {0x104101c80, 0x14000a507c0})
    github.com/hashicorp/terraform-provider-vault/internal/provider/provider.go:259 +0x70
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0x1400019dce0, {0x1041c0f58, 0x14000c04450}, 0xd?, {0x104101c80, 0x14000a507c0})
    github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:795 +0xe8
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).ReadDataApply(0x1400019dce0, {0x1041c0f58, 0x14000c04450}, 0x14000c30000, {0x104101c80, 0x14000a507c0})
    github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:1015 +0x124
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadDataSource(0x14000864b10, {0x1041c0f58?, 0x14000c04360?}, 0x14000916020)
    github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/grpc_provider.go:1251 +0x2e0
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadDataSource(0x14000a8fd60, {0x1041c0f58?, 0x140008692c0?}, 0x14000066780)
    github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:699 +0x324
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadDataSource_Handler({0x104159b60?, 0x14000a8fd60}, {0x1041c0f58, 0x140008692c0}, 0x14000b2ed80, 0x0)
    github.com/hashicorp/[email protected]/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:539 +0x164
google.golang.org/grpc.(*Server).processUnaryRPC(0x140002ae800, {0x1041c0f58, 0x14000869230}, {0x1041c7340, 0x14000182680}, 0x140009aec60, 0x14000868bd0, 0x104c6e488, 0x0)
    google.golang.org/[email protected]/server.go:1372 +0xb8c
google.golang.org/grpc.(*Server).handleStream(0x140002ae800, {0x1041c7340, 0x14000182680}, 0x140009aec60)
    google.golang.org/[email protected]/server.go:1783 +0xc44
google.golang.org/grpc.(*Server).serveStreams.func2.1()
    google.golang.org/[email protected]/server.go:1016 +0x5c
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 26
    google.golang.org/[email protected]/server.go:1027 +0x138

Error: The terraform-provider-vault_v3.24.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

Terraform Configuration Files

terraform {
  required_providers {
    vault = {
      source  = "hashicorp/vault"
      version = "3.24.0"
    }
  }

  required_version = "1.7.0"
}

provider "vault" {
  address = "https://..."
  auth_login_oidc {
    role = "default"
  }
}

data "vault_kv_secret_v2" "some_secret" {
  mount = "..."
  name  = "..."
}
data "vault_kv_secret_v2" "some_other_secret" {
  mount = "..."
  name  = "..."
}

Steps to Reproduce

1. Use provider with auth_login_oidc
2. Declare at least two data sources of type vault_kv_secret_v2. If you use just one, it does work, so it's likely some race condition.
3. Run terraform apply or terraform plan
4. Browser gets opened to log you in and the session is expired (so SSO does not automatically redirect you back)
5. Do nothing there
6. After 2 minutes, the provider crashes in a nasty way

Debug Output

No response

Panic Output

No response

Important Factoids

Users are very likely to issue terraform apply and miss the fact they had a new browser tab opened - it can mix up with the time they opened a new tab or for larger deployments they can just go to grab a coffee.

References

No response

Would you like to implement a fix?

None

[danielskowronski]

The logic in https://github.com/hashicorp/terraform-provider-vault/blob/main/internal/provider/meta.go#L268 seems to have some bug. Likely, the cloned client is left in an unclean state after timeout from TF (registered handler for /oidc/callback) and there's some missing check to see if it can be re-used so an unsafe authLogin.Login(clone) is issued.

Or it can be caused by the library used, https://github.com/hashicorp/vault-plugin-auth-jwt/pull/115 seems to confirm this behaviour.

[fairclothjm]

@danielskowronski Hi, thanks for reporting! I believe this does need to be fixed in vault-plugin-auth-jwt. We will see if we can get this resolved.