Private cluster contacted on localhost

[tback]

trafficstars

TL;DR

I created a private cluster wie the official private-cluster and auth modules. After updating the modules terraform tries to connect to it on localhost.

Expected behavior

I expect terraform to authenticate against the cluster and correctly and connect to the privtate cluster on it's public api endpoint.

Observed behavior

Error: Get "http://localhost/api/v1/namespaces/redacted": dial tcp [::1]:80: connect: connection refused
    with kubernetes_namespace.redacted,
    on kubernetes.tf line 99, in resource "kubernetes_namespace" "tls":
    line: resource "kubernetes_namespace" "redacted" {

Terraform Configuration

provider "helm" {
  kubernetes {
    cluster_ca_certificate = module.gke_auth.cluster_ca_certificate
    host                   = module.gke_auth.host
    token                  = module.gke_auth.token
  }
}

provider "kubernetes" {
  cluster_ca_certificate = module.gke_auth.cluster_ca_certificate
  host                   = module.gke_auth.host
  token                  = module.gke_auth.token
}

module "gke_auth" {
  source = "terraform-google-modules/kubernetes-engine/google//modules/auth"

  project_id   = google_project.REDACTED.project_id
  cluster_name = local.cluster_name
  location     = local.region

  depends_on = [
    google_project.REDACTED
  ]
}

module "gke_REDACTED" {
  source  = "terraform-google-modules/kubernetes-engine/google//modules/private-cluster"
  version = "~> 35.0"

  project_id = google_project.REDACTED.project_id
  name       = local.cluster_name

  regional = true
  region   = local.region
  zones    = local.zones

  enable_private_nodes = true

  network            = google_compute_network.REDACTED.name
  subnetwork         = google_compute_subnetwork.REDACTED.name
  ip_range_pods      = "pods"
  ip_range_services  = "services"
  initial_node_count = 1

  http_load_balancing             = false
  network_policy                  = true
  remove_default_node_pool        = true
  horizontal_pod_autoscaling      = true
  enable_vertical_pod_autoscaling = true
  logging_service                 = "none"
  monitoring_service              = "none"

  depends_on = [
    module.project_services,
    module.cloud_router,
  ]

  grant_registry_access = true
  registry_project_ids = [
    "REDACTED",
  ]

  node_pools = [
    {
      name               = "default-node-pool"
      machine_type       = "e2-standard-8"
      node_locations     = join(",", local.zones)
      min_count          = 1
      max_count          = 10
      local_ssd_count    = 0
      disk_size_gb       = 100
      disk_type          = "pd-standard"
      image_type         = "COS_CONTAINERD"
      enable_gcfs        = false
      auto_repair        = true
      auto_upgrade       = true
      preemptible        = false
      initial_node_count = 1
    },
  ]

  node_pools_metadata = {
    all = {
      block-project-ssh-keys = true
    }
  }

}

Terraform Version

Terraform v1.10.4
on darwin_arm64
...
+ provider registry.terraform.io/hashicorp/google v6.15.0
+ provider registry.terraform.io/hashicorp/google-beta v6.15.0
+ provider registry.terraform.io/hashicorp/helm v2.17.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.35.1
...
+ provider registry.terraform.io/mongodb/mongodbatlas v1.25.0
...

Additional information

The cluster has been running for about two years. GKE, terraform, terraform modules have been updated multiple times.

[tback]

related to https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/issues/1675

[tback]

I finally found a workaround after reading https://stackoverflow.com/a/76219587/246241. A partial apply on the module fixes this the time being (probably until the next change on the cluster):

terraform apply -target module.gke_REDACTED

[apeabody]

Thanks @tback - I believe this is also aligned with https://github.com/hashicorp/terraform-provider-kubernetes/issues/1479 which we have been tracking for a while.

I wonder if an explicit dependency on the cluster module could help, have you tried something like this?

module "gke_auth" {
  source = "terraform-google-modules/kubernetes-engine/google//modules/auth"

  project_id   = google_project.REDACTED.project_id
  cluster_name = module.gke_REDACTED.name
  location     = module.gke_REDACTED.location

  depends_on = [
    google_project.REDACTED
  ]
}

Even further, you probably don't need to use the gke_auth module unless you are generating a kubeconfig file. Here is a possible example:

data "google_client_config" "default" {}

provider "kubernetes" {
  host                   = "https://${module.gke_REDACTED.endpoint}"
  token                  = data.google_client_config.default.access_token
  cluster_ca_certificate = base64decode(module.gke_REDACTED.ca_certificate)
}