kubernetes_manifest does not address credentials properly

[vladimirtiukhtin]

Terraform Version, Provider Version and Kubernetes Version

Terraform version:
Terraform v1.4.6
on linux_amd64
Kubernetes provider version:
+ provider registry.terraform.io/hashicorp/kubernetes v2.21.1
Kubernetes version:
1.24

Affected Resource(s)

• kubernetes_manifest

Behavior

I have two kubernetes providers defined: one (vanilla) consumes credentials from environment variables through KUBE_TOKEN, another (EKS) uses aws_eks_cluster_auth resource to obtain credentials. And it happily works with all kubernetes resources except kubernetes_manifest. It seems to grab access_token from environment variable instead of one configured in the provider definition

provider "kubernetes" {
  alias                  = "aws"
  host                   = data.aws_eks_cluster.environment_cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.environment_cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.environment_cluster.token
}

So I very much understandably get Unauthorized unless I run unset KUBE_TOKEN which is not an acceptable solution because provider one needs it

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jrhouston]

Looks like this is a bug in the initialization logic for the manifest part of the provider. For resources that aren't kubernetes_manifest we use a DefaultFunc to set the value from an environment variable. So if a value is set in the config, it will ignore the environment variable.

However, in the manifest code we had to write custom logic to deal with environment variables (because it uses terraform-plugin-go and not the SDK) which actually does the inverse of this. So if the environment variable is set, it will override what's specified in the config.

This is a bug and we'll fix this to make it consistent with what the SDK does.

[sczizzo]

Running into this myself, now:

% terraform version
Terraform v1.6.5
on linux_amd64
+ provider registry.terraform.io/hashicorp/kubernetes v2.24.0

Very similar setup: I want to use two separate, aliased Kubernetes providers. Works great, except the kubernetes_manifest resources are picking up my default kubeconfig from my environment rather than what's in the provider configuration.