GKE Sandbox configuration

[vjoshi-hub]

I am trying to create a cluster and nodepool with image_type as COS_CONTAINERD and node version as 1.14.8-gke.17 and sandbox_enabled as true. I am using submodule private-beta-cluster. The cluster gets created with no errors but in the console nodepool configuration shows Sandbox with gVisor as disabled. I am not sure if I am missing something.

[morgante]

Can you share your module config and terraform version? Thanks!

[vjoshi-hub]

gke.txt Terraform v0.12.13

• provider.google v2.18.1
• provider.google-beta v2.18.1

[vjoshi-hub]

I have upgraded my provider versions - Terraform v0.12.13

• provider.google v2.19.0
• provider.google-beta v2.19.0 But still facing same issue with CFT module. However, I am able to create Cluster and Node pool with GKE sandbox enabled using terraform resource. I used google-beta provider. gke-terraform.txt

[vjoshi-hub]

I am creating additional nodepool with sandbox_enabled = true and image_type ="COS_CONTAINERD". It creates cluster and nodepool successfully. However, nodepool shows correct image type but GKE sandbox disabled.

[morgante]

You actually have to set the sandbox_enabled variable to true on the module itself, not for the individual node pool. Like this example. Can you change that and confirm it works?

[vjoshi-hub]

I moved sandbox_enabled = true in module. I have image_type = "COS_CONTAINERD" in node pool configuration. However, I got error as below - Error: googleapi: Error 400: Gvisor sandbox type is only available on the COS_CONTAINERD image family., badRequest

on .terraform/modules/gke_cluster_hmt_prod/terraform-google-modules-terraform-google-kubernetes-engine-f99ddf4/modules/beta-private-cluster/cluster.tf line 22, in resource "google_container_cluster" "primary": 22: resource "google_container_cluster" "primary" {

Also I am using terraform-google-modules/kubernetes-engine/google//modules/beta-private-cluster module. Example mentioned above is using beta-public-cluster.

[morgante]

Can you provide the output of terraform plan?

There shouldn't be a difference between private and public clusters when it comes to sandboxing.

[vjoshi-hub]

terraform-plan.txt

[morgante]

Thanks, I think the issue is related to the embedded default node pool which we should probably remove entirely. We're going to have to do a bit more digging on this unfortunately.

[robertb724]

@morgante I have been trying to use this with the safer-cluster module. It appears the issue is due to the fact that this setting is meant to be enabled on a per node-pool setting while in the module it is set at the cluster level and applied to all node-pools. In the sandbox docs it states that there must be at least one node pool with the sandbox disabled.

https://cloud.google.com/kubernetes-engine/docs/concepts/sandbox-pods#limitations-nodepool

[morgante]

Thanks for digging into it. It looks like we will indeed need to add a per-node-pool setting for sandboxing since we can't always rely on the default node pool.

[iamasmith]

The GKE Sandbox notes say that you need another nodepool without sandboxing to be able to use it, as the default node pool would be the first up I think this is a constraint from Google.

"When using GKE Sandbox, your cluster must have at least two node pools. You must always have at least one node pool where GKE Sandbox is disabled. This node pool must contain at least one node, even if all your workloads are sandboxed."

https://cloud.google.com/kubernetes-engine/docs/concepts/sandbox-pods