Unable to use the new DNS endpoint exclusively.

[TheKangaroo]

TL;DR

My goal is to use only the DNS endpoint on my GKE clusters:

This is not possible at the moment because I need to set:

  enable_private_endpoint       = true
  deploy_using_private_endpoint = true

for the DNS endpoint to be enabled. However, enable_private_endpoint will create an empty master_authorized_networks_config {} block in this line, and applying this will re-enable access via IPv4.

I created a PR to fix this, but I'm not 100% sure if it breaks anything unrelated to DNS endpoints.

Another thing I observed while experimenting with this setting is that removing master_authorized_networks_config {} won't actually change my cluster to disable IPv4 access. The code seems to work for both cases: with master_authorized_networks_config {} and IPv4 enabled, and omitted with IPv4 disabled. Maybe this is just a limitation in the API, where changing master_authorized_networks_config {} to (empty) does not trigger an update. 🤔

Expected behavior

No response

Observed behavior

No response

Terraform Configuration

n.a.

Terraform Version

tofu version                                                                                          app-publisher-dev-7ed19f10be
OpenTofu v1.8.7
on darwin_arm64
+ provider registry.terraform.io/hashicorp/google v4.84.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.23.0
+ provider registry.terraform.io/hashicorp/random v3.5.1

Additional information

No response

[apeabody]

Thanks @TheKangaroo - Interesting, it looks like the presence of the (empty) master_authorized_networks_config block is actually being used as a substitute for enabled: https://github.com/hashicorp/terraform-provider-google/blob/main/google/services/container/resource_container_cluster.go#L5035

[apeabody]

Interesting @TheKangaroo - So currently the module's enable_private_endpoint actuates the provider's private_cluster_config.enable_private_endpoint which actuates the API's PrivateClusterConfig.enablePrivateEndpoint which is deprecated. The recommendation is to use ControlPlaneEndpointsConfig.IPEndpointsConfig.enable_public_endpoint, however ControlPlaneEndpointsConfig.IPEndpointsConfig.enabled is currently hardcoded to true in the Provider.

As masterAuthorizedNetworksConfig is part of ControlPlaneEndpointsConfig.IPEndpointsConfig, if masterAuthorizedNetworksConfig is created using the Provider, it is likely resulting in the side-effect of ControlPlaneEndpointsConfig.IPEndpointsConfig.enabled = true.

[apeabody]

Here is a PR to add Provider support for ControlPlaneEndpointsConfig.IPEndpointsConfig.enabled: https://github.com/hashicorp/terraform-provider-google/issues/20369

[TheKangaroo]

Ah, I think I understand now. Thanks for reviewing the current implementation and providing the explanation. Since we can already use DNS endpoints and disabling IP endpoints would just be an added benefit, I'm fine with waiting for https://github.com/hashicorp/terraform-provider-google/issues/20369.

[asafhm]

Here is a PR to add Provider support for ControlPlaneEndpointsConfig.IPEndpointsConfig.enabled: hashicorp/terraform-provider-google#20369

That provider's PR is merged. Can we fix this in the module?

[TheKangaroo]

I haven't been able to find the time to work on the PR since then, but I will do so as soon as possible.

[TheKangaroo]

I just checked and saw that @apeabody already improved the dns endpoint config a couple of days ago (https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/commit/1c6ff12b9c01fe77633bae942e9c8ff8ff38f17a). Is my understanding correct, that after these changes we only need to add something like:

   dynamic "control_plane_endpoints_config" {
-    for_each = var.dns_allow_external_traffic != null ? [1] : []
+    for_each = var.dns_allow_external_traffic != null || var.master_authorized_networks == null ? [1] : []
     content {
+      ip_endpoints_config {
+        enabled = var.master_authorized_networks == null ? false : true
+      }
       dns_endpoint_config {
         allow_external_traffic = var.dns_allow_external_traffic
       }
    }
  }

and let master_authorized_networks default to null

   variable "master_authorized_networks" {
     type        = list(object({ cidr_block = string, display_name = string }))
     description = "List of master authorized networks. If none are provided, disallow external access (except the cluster node IPs, which GKE automatically whitelists)."
-    default     = []
+    default     = null
   }

to actually disable ip endpoints on empty master_authoritzed_networks?