Creation of kube-dns cm fails on newly created clusters

[grieshaber]

TL;DR

Creating a new cluster with this module and configuring the use of stub_domains fails with:

Error: The configmap "kube-dns" does not exist
│ 
│   with module.gke.module.gke.kubernetes_config_map_v1_data.kube-dns[0],
│   on .terraform/modules/gke.gke/modules/private-cluster/dns.tf line 23, in resource "kubernetes_config_map_v1_data" "kube-dns":
│   23: resource "kubernetes_config_map_v1_data" "kube-dns" {

Expected behavior

The kube-dns config-map will be adapted properly, right after the cluster and node-pool are provisionend successfully.

Observed behavior

Adaption of the configMap fails because terraform fails accessing the actual configMap within the cluster.

Terraform Configuration

provider "kubernetes" {
  host                   = "https://${module.gke.cluster_endpoint}"
  token                  = data.google_client_config.default.access_token
  cluster_ca_certificate = base64decode(module.gke.cluster_ca_crt)
}

provider "google" {
  project = var.project_id
  region  = var.region

  credentials = file("./creds/serviceaccount.json")
}

module "gke" {
  source  = "gitlab.service.example.de/software/terraform-google-onedata/gke"
  ... 

}

# This module above is created by us and contains the following relevant snippet
module "gke" {
  depends_on = [
    google_compute_subnetwork.cluster,
    google_compute_network.cluster,
    google_dns_managed_zone.dns_zone
  ]
  source     = "terraform-google-modules/kubernetes-engine/google//modules/private-cluster"
  version    = "21.1.0"
  project_id = var.project_id
  ... 
  stub_domains = { "example.de" : ["10.0.0.3"] }
  ... 
}

Terraform Version

Terraform v1.1.7
Google Provider 4.24.0

Additional information

No response

[bharathkkb]

@grieshaber Is this a sequencing issue where we are trying to access the CM before it is created or is the kube-dns CM not in the cluster at all? Could you post a complete sample config I can use to reproduce? /cc @apeabody who touched this recently incase you had any thoughts.

[apeabody]

@grieshaber Thanks for reaching out, can you also let us know the version of your Kubernetes Provider? e.g 2.11.0?

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days

[rpere]

@grieshaber Is this a sequencing issue where we are trying to access the CM before it is created or is the kube-dns CM not in the cluster at all? Could you post a complete sample config I can use to reproduce? /cc @apeabody who touched this recently incase you had any thoughts.

Seeing the exact same issue for a brand new cluster setup using module :

  source                     = "terraform-google-modules/kubernetes-engine/google//modules/beta-private-cluster-update-variant"
  version                    = "23.3.0"

and following providers : kubernetes v2.15.0 random v3.4.3 google v4.43.1

kube-dns CM exists in the cluster in the kube-sytem ns

@grieshaber what did you end up doing here ?

[cread]

I'm also seeing this error.

Terraform Version: v1.4.6

Module Version: v26.1.1

Providers Versions:

• google: 4.66.0
• google-beta: 4.66.0
• kubernetes: 2.20.0

[phandox]

I also see this error, but it's on already created cluster. I added just upstream_nameservers option to the module invocation.

Terraform v1.3.9 on linux_amd64

• provider registry.terraform.io/hashicorp/google v4.67.0
• provider registry.terraform.io/hashicorp/google-beta v4.67.0
• provider registry.terraform.io/hashicorp/kubernetes v2.20.0
• provider registry.terraform.io/hashicorp/random v3.3.2

module.gke_dev-v2.kubernetes_config_map_v1_data.kube-dns-upstream-namservers[0]: Creating...
╷
│ Error: The configmap "kube-dns" does not exist
│ 
│   with module.gke_dev-v2.kubernetes_config_map_v1_data.kube-dns-upstream-namservers[0],
│   on .terraform/modules/gke_dev-v2/modules/beta-private-cluster/dns.tf line 45, in resource "kubernetes_config_map_v1_data" "kube-dns-upstream-namservers":
│   45: resource "kubernetes_config_map_v1_data" "kube-dns-upstream-namservers" {
│ 
╵

The ConfigMap is present in cluster

k get cm -n kube-system kube-dns -o yaml
apiVersion: v1
kind: ConfigMap
metadata:
  creationTimestamp: "2022-04-04T10:02:38Z"
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
  name: kube-dns
  namespace: kube-system
  resourceVersion: "384016010"
  uid: bdee174a-23a4-4571-bcad-b20a0cc838da