Terraform Kubernetes Provider Fails to Connect to EKS Cluster in GitLab CI Pipeline: "dial tcp [::1]:80: connect: connection refused" [closed]

[nibrazrehaman]

Description

When the pipeline runs terraform apply, it fails with the following error:

Error: Get "http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth": dial tcp [::1]:80: connect: connection refused

`with kubernetes_config_map_v1_data.aws_auth, on aws_auth.tf line 1, in resource "kubernetes_config_map_v1_data" "aws_auth": 1: resource "kubernetes_config_map_v1_data" "aws_auth" {

Error: Get "http://localhost/api?timeout=32s": dial tcp [::1]:80: connect: connection refused

with kubernetes_env.aws_node, on k8s.tf line 2, in resource "kubernetes_env" "aws_node": 2: resource "kubernetes_env" "aws_node" {`

Terraform, AWS provider and Kubernetes version

Terraform version:  1.6.4
Aws Provider version: 5.84.0
Kubernetes version: 2.35.1

Terraform configuration

provider "kubernetes" {
  host                   = data.aws_eks_cluster.gis.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.gis.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.gis.token
}

data "aws_eks_cluster" "gis" {
  name = "my-cluster-name"
}
 
data "aws_eks_cluster_auth" "gis" {
  name = data.aws_eks_cluster.gis.name
}

Question

1. What could be causing the Kubernetes provider to fail during the EKS upgrade from version 1.27 to 1.28?
2. Are there any additional configurations or Terraform-specific steps required to ensure compatibility and a successful upgrade in the GitLab CI pipeline?

Steps to Reproduce

1. Use the provided Terraform configuration.
2. Initiate an EKS cluster upgrade from version 1.27 to 1.28 in a GitLab CI pipeline.
3. Observe the error during the terraform apply step.

Expected Behavior

1. The Kubernetes provider should connect to the correct EKS cluster endpoint during the upgrade.
2. The EKS cluster upgrade process should complete successfully without errors.
3. Any insights into resolving this issue would be greatly appreciated!

Thank you in advance for your help!

[CrashLaker]

facing a similar scenario here.

i was suspecting this was because i had kubernetes providers being set outside and inside a module.

in my case my terraform run inside a pod in another eks cluster and it assumes a role to connect to my eks.

sometimes i get this localhost and others an weird permission error indicating that the pod running the terraform cannot access the configmap. i think this shouldn't be happening since i'm setting the provider inside kubernetes_config_map_v1_data.

regards,c.

[GolubevV]

I think we have exactly the same problem since we moved TFE to run inside Kubernetes. We are using community terraform-aws-eks module of version 18.31.2 and here is the error:

Error: configmaps "aws-auth" is forbidden: User "system:serviceaccount:terraform-enterprise-agents:default" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
with module.eks.module.eks.kubernetes_config_map_v1_data.aws_auth[0]
on .terraform/modules/eks.eks/main.tf line 475, in resource "kubernetes_config_map_v1_data" "aws_auth":

I suspect this is a regression of what was supposed to be fixed 5 years ago in https://github.com/hashicorp/terraform-provider-kubernetes/pull/690

e.g - when terraform runtime is located inside K8s cluster, it assumes the identity based on the pod instead of using explicitly provided configuration with token, url etc.