kubernetes_manifest might not be respecting `KUBE_PROXY_URL`

[kriswuollett]

I suspect that operations that implement the kubernetes_manifest is attempting to connect to non-routable IP from my local machine by not using the port-forwarded https proxy that I defined with KUBE_PROXY_URL. Creating a kubernetes_namespace with this setup works fine. My module with kubernetes_manifest resources works fine in a development cluster that has publicly accessible nodes and API endpoint without a KUBE_PROXY_URL defined.

I guess the workaround I'll have to try is converting the manifests to HCL.

Terraform Version, Provider Version and Kubernetes Version

Terraform v1.2.2
on darwin_amd64
+ provider registry.terraform.io/hashicorp/google v4.22.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.11.0

Affected Resource(s)

• kubernetes_manifest

Terraform Configuration Files

Relevant configuration section:

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "4.22.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "2.11.0"
    }
  }
  backend "gcs" {
    bucket = "MY_BUCKET"
    prefix = "MY_CLUSTER/CLUSTER_SERVICES"
  }
}

data "google_client_config" "provider" {}

provider "kubernetes" {
  host                   = var.kubernetes_host
  token                  = data.google_client_config.provider.access_token
  cluster_ca_certificate = base64decode(var.kubernetes_host_ca_certificate)
}

Steps to Reproduce

1. Create a Kubernetes cluster that is not accessible from the Internet like a Google Cloud cluster with private nodes and private endpoint.
2. Create a bastion host in a subnet that is authorized to access the Kubernetes endpoint. Bastion runs a http proxy like tinyproxy.
3. Open up a SSH connection to the bastion with port forwarding to the private endpoint (e.g., -L 8888:localhost:8888)
4. Run Terraform like KUBE_PROXY_URL="http://localhost:8888" terraform ....

Expected Behavior

Terraform should be able to create all kinds of kubernetes resources.

Actual Behavior

I can create a kubernetes_namespace, but when setting up third-party services like cert-manager through kubernetes_manifest resources that created with yamldecode(file(...)), I get errors like the following:

2022-06-03T16:08:49.216-0500 [DEBUG] provider.terraform-provider-kubernetes_v2.11.0_x5: 2022-06-03T16:08:49.157-0500 [DEBUG] [InvalidClientConfiguration]: Config="&rest.Config{Host:\"https://[[PRIVATE_K8S_IP]]\", APIPath:\"\", ContentConfig:rest.ContentConfig{AcceptContentTypes:\"\", ContentType:\"\", GroupVersion:(*schema.GroupVersion)(nil), NegotiatedSerializer:(*serializer.negotiatedSerializerWrapper)(0xc001bfb5e0)}, Username:\"\", Password:\"\", BearerToken:\"--- REDACTED ---\", BearerTokenFile:\"\", Impersonate:rest.ImpersonationConfig{UserName:\"\", UID:\"\", Groups:[]string(nil), Extra:map[string][]string(nil)}, AuthProvider:<nil>, AuthConfigPersister:rest.AuthProviderConfigPersister(nil), ExecProvider:<nil>, TLSClientConfig:rest.sanitizedTLSClientConfig{Insecure:false, ServerName:\"\", CertFile:\"\", KeyFile:\"\", CAFile:\"\", CertData:[]uint8(nil), KeyData:[]uint8(nil), CAData:[]uint8{[[REDACTED]]}, NextProtos:[]string(nil)}, UserAgent:\"\", DisableCompression:false, Transport:http.RoundTripper(nil), WrapTransport:(transport.WrapperFunc)(nil), QPS:0, Burst:0, RateLimiter:flowcontrol.RateLimiter(nil), WarningHandler:rest.WarningHandler(nil), Timeout:0, Dial:(func(context.Context, string, string) (net.Conn, error))(nil), Proxy:(func(*http.Request) (*url.URL, error))(nil)}"
2022-06-03T16:08:49.217-0500 [ERROR] vertex "module.cluster-services.module.cert_manager.module.stage_1.kubernetes_manifest.file[\"manifests/ClusterRoleBinding-cert-manager-cainjector.yml\"]" error: Invalid configuration for API client

References

• similar to #1653, but it is clear that using the proxy works since creating and refreshing kubernetes_namespace resources works fine
• bastion discussion in #812

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[kriswuollett]

This may the case that I'm still using a data source to configure the provider since this has to occur during the plan stage? Will close as a duplicate if so once I get a chance to check.

[kriswuollett]

I'm still encountering the issue specifically with kubernetes_manifest resources. Existing resources in state like kubernetes_namespace that are already in Terraform state refresh okay before the error occurs. I changed my Kubernetes provider config to the following to avoid issues with provider dependencies during the planning stage:

provider "kubernetes" {
  config_path    = "~/.kube/config"
  config_context = var.kubernetes_context
}

[geNAZt]

We also hit this issue with basically the same setup. We also use a GKE private cluster and IAP tunnels to SSH on a bastion with tinyproxy on it. It works for everything non kubernetes_manifest related. But when it tries to apply kubernetes_manifest it tries to directly talk to the host URL and not the configured KUBE_PROXY_URL.

[mikeroll]

Just encountered this too. The curious thing is, while KUBE_PROXY_URL doesn't take effect for kubernetes_manifest, proxy_url in the provider configuration block does. Judging by way the code is structured, kubernetes_manifest seems to use its own sub-provider with a separate configuration code path, so it must be somehow out of sync with that of the main provider.

Lookups for both proxy_url and KUBE_PROXY_URL go together so I have a hard time pinpointing the issue. https://github.com/hashicorp/terraform-provider-kubernetes/blob/73e25581e38a11e337bc8f01c1d15f2e4c6f961f/manifest/provider/configure.go#L492-L508