Strange issue with aws-auth "connection refused" only when updating AWS tags

[rsmithcooper]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: 1.42
Kubernetes provider version: 2.21.1
Kubernetes version: 1.27

Affected Resource(s)

• kubernetes_config_map_v1_data

Terraform Configuration Files

### LOCALS ###
locals {
...
  # CHANGING THE TAGS SOMEHOW BREAKS THE TF. AWS-AUTH CANNOT CONNECT. VERY STRANGE
  tags = {
    Application = local.name
    ManagedBy   = "terraform"
    Environment = "dev"
    Repository  = var.repository
    # Example = ""
  }
}

### EKS ###
resource "aws_eks_cluster" "this" {
 tags = local.tags
...
}

resource "aws_eks_node_group" "this" {
  cluster_name    = aws_eks_cluster.this.name

tags = local.tags
...
}

### KUBERNETES ###

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.this.token
}

locals {
  aws_auth_map_roles = [
    {
      rolearn  = aws_iam_role.cluster["WorkerRole"].arn
      username = "system:node:{{EC2PrivateDNSName}}"
      groups = [
        "system:bootstrappers",
        "system:nodes",
        "system:masters"
      ]
    },
    {
      rolearn  = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/${element(tolist(data.aws_iam_roles.admin.names), 0)}"
      username = "admin"
      groups = [
        "system:bootstrappers",
        "system:nodes",
        "system:masters"
      ]
    }
  ]

  aws_auth_configmap_data = {
    mapRoles = yamlencode(local.aws_auth_map_roles)
  }
}

#update the kubeconfig
resource "null_resource" "set_kubeconfig" {
  provisioner "local-exec" {
    command = "aws eks --region ${var.aws_region} update-kubeconfig --name ${local.name}-cluster"
  }
  depends_on = [aws_eks_cluster.this, aws_security_group.alb_internal]
}

#update aws-auth
resource "kubernetes_config_map_v1_data" "aws_auth" {
  force = true

  metadata {
    name      = "aws-auth"
    namespace = "kube-system"
  }

  data = local.aws_auth_configmap_data

  depends_on = [aws_eks_addon.this, null_resource.set_kubeconfig]
}

Debug Output

could not set this correctly to get only the data I wanted to share

Panic Output

https://gist.github.com/lfc550/fe80b72a6022a6f30d651fe03a35b9cd

Steps to Reproduce

1. change contents of aws-auth using terraform resourcew
2. terraform apply
3. Confirm that it worked fine
4. change any local.tags value
5. terraform apply
6. will fail with above error
7. change tags back to what it was
8. terraform apply
9. works fine

Expected Behavior

Changing tags of my AWS resources should not affect the ability to connect to aws-auth

Actual Behavior

Changing tags gives me the connection refused error. if I change anything else, including the aws-auth resource, it works fine.

Important Factoids

Honestly, the key fact is that it works perfectly fine OTHER THAN when changing AWS tags

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[alexsomesan]

Are you creating the EKS cluster in the same apply operation where this error occurs?

If so, that is not a supported use-case, because Terraform cannot guarantee that the values of the attributes in the provider "kubernetes" {...} block are actually available from the aws_eks_cluster resource early enough to be used for configuring the Kubernetes provider. As you can see from the error message you shared in the gist, the provider did not receive a valid API URL value for the host attribute.

Also, is this the exact configuration that led to this error? I'm not seeing any data source called aws_eks_cluster although it's being referenced by the kubernetes provider attributes?

[rsmithcooper]

I am creating the cluster in the same apply, yes. I still don't understand how changing an AWS tag has any downstream implications for connecting to the cluster - can you explain that in more detail? Also, is there a workaround for this in any way outside of splitting up my state file?

Below is from my data.tf

data "aws_eks_cluster_auth" "this" { name = aws_eks_cluster.this.name }

[koslowdavida]

Having this exact issue; @alexsomesan are we supposed to create a separate state that refers to the "eks" module from the first state and have the separate state manage stuff like node groups?

[alexsomesan]

separate

The minimum requirement is that the aws_eks_cluster resource be managed in a separate state. Node groups are not a requirement, only the API server is.