Support for configuring Hyperdisk provisioned Throughput and IOPS

[FearlessHyena]

TL;DR

Since Hyperdisks allow you to specify the Throughput and IOPS that should be provisioned when creating them, it would be great if the node_pools property added support for them

Terraform Resources

Detailed design

Two new variables under the `node_pools` property would be needed which would only be used in case of Hyperdisks and ignored for Persistent disks. The [google_compute_disk resource] (https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_disk) can be used for reference
- disk_provisioned_iops
- disk_provisioned_throughput

There is also the [enable_confidential_compute](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_disk#enable_confidential_compute-1) property but not sure if it would be in scope of this feature request

Additional information

No response

[DrFaust92]

https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/hyperdisk#create according to docs its not directly related to the GKE config and needs to be done by creating a relevant storage class and setting it on relevant PVC:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: balanced-storage
provisioner: pd.csi.storage.gke.io
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
  type: hyperdisk-balanced
  provisioned-throughput-on-create: "250Mi"
  provisioned-iops-on-create: "7000"

and

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: podpvc
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: balanced-storage
  resources:
    requests:
      storage: 20Gi

and using that for your workloads the above may differ based on your config.

as this is not a module issue I think it cant be closed

[FearlessHyena]

Certain newer GCP instance types like the C4/C4D/C4A only support Hyperdisk volumes so when you create a nodepool with any of those instance types, the disk_type needs to be set to a hyperdisk variant like hyperdisk-balanced

I believe you don't need a PVC if you plan to use the Hyperdisk as ephemeral storage, as GKE automatically reserves part of the boot disk to serve as ephemeral storage for Pods Unfortunately unless the Throughput/IOPS are explicitly specified, the default values are used which isn't ideal for many production environments

[marinflorin]

Hey @FearlessHyena, it looks like the documentation needs to be updated, but there is no validation against the disk_type. You can go ahead and use hyperdisk-balanced. I gave it a try, and it worked. Also, below, you can find an example that is using hyperdisk-balanced.

https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/blob/main/examples/confidential_safer_cluster/main.tf#L102

[voron]

There is no such option in GKE node pool API support to configure disk IOPS/throughput for the node's main/boot disk. Thus I don't expect this feature to appear anytime soon. Possible work-around is to use daemon set on the corresponding node pool to decrease provisioned IOPS/throughput to free values in runtime.

[DrFaust92]

seems that its supported via https://github.com/GoogleCloudPlatform/magic-modules/pull/14600/files

[kartoch]

1. disk_size is not taking into account.

For the following configuration:

  node_pools = [
    {
      name               = "arm-pool"
      machine_type       = "c4a-standard-1"
      initial_node_count = 1
      min_count          = 1
      max_count          = 5
      autoscaling        = true
      disk_size          = 20
      disk_type          = "hyperdisk-balanced"
    }
  ]

Each node has a 100GB disk not 20GB (i tried other values but no changes)

2. default values of provisioned IOPS and provisioned throughput can be costly / are too high

Each hypedisk has 3,600 provisioned IOPS and 290 MB/s provisioned throughput by default via the terraform module.

Quote from the disk pricing page:

"Hyperdisk Balanced and Hyperdisk Balanced High Availability charge a monthly rate for the provisioned IOPS and provisioned throughput (in MBps) in excess of the baseline values of 3,000 IOPS and 140 MBps throughput".

This means there are 80GB extra capacity, 600 extra provisioned IOPS and 150 MB/s extra if I want minimal setup to limit extra charges.

[voron]

seems that its supported via https://github.com/GoogleCloudPlatform/magic-modules/pull/14600/files

I don't see the support in terraform GKE modules yet, but it's good to know things are done on GCP provider side.