Toggling 'allow_volume_expansion' on 'storage_class' resources have no effect

[abstrask]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: 1.0.1
Kubernetes provider version: 2.3.2 (also tried with 1.6.x and 2.0.x)
Kubernetes version: 1.20 (AWS EKS)

Affected Resource(s)

• storage_class

Terraform Configuration Files

resource "kubernetes_storage_class" "csi-gp2" {
  metadata {
    name = "csi-gp2"
    annotations = {
      "storageclass.kubernetes.io/is-default-class" = "true"
    }
  }
  storage_provisioner    = "ebs.csi.aws.com"
  reclaim_policy         = "Delete"
  volume_binding_mode    = "WaitForFirstConsumer"
  allow_volume_expansion = "true"
  parameters = {
    type = "gp2"
  }

}

Debug Output

2021-07-06T10:25:55.859Z [INFO]  plugin.terraform-provider-kubernetes_v2.3.2_x5: 2021/07/06 10:25:55 [INFO] Submitted updated storage class: &v1.StorageClass{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"csi-gp2", GenerateName:"", Namespace:"", SelfLink:"", UID:"e29db48f-c27a-4c7d-a806-accf17e622d6", ResourceVersion:"3519662", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63759526795, loc:(*time.Location)(0x325c660)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"storageclass.kubernetes.io/is-default-class":"true"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kubectl-patch", Operation:"Update", APIVersion:"storage.k8s.io/v1", Time:(*v1.Time)(0xc00000d830), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00000d848)}, v1.ManagedFieldsEntry{Manager:"HashiCorp", Operation:"Update", APIVersion:"storage.k8s.io/v1", Time:(*v1.Time)(0xc00000d860), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00000d878)}, v1.ManagedFieldsEntry{Manager:"kubectl-edit", Operation:"Update", APIVersion:"storage.k8s.io/v1", Time:(*v1.Time)(0xc00000d890), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00000d8a8)}}}, Provisioner:"ebs.csi.aws.com", Parameters:map[string]string{"type":"gp2"}, ReclaimPolicy:(*v1.PersistentVolumeReclaimPolicy)(0xc001187170), MountOptions:[]string(nil), AllowVolumeExpansion:(*bool)(0xc00118a5ee), VolumeBindingMode:(*v1.VolumeBindingMode)(0xc001187180), AllowedTopologies:[]v1.TopologySelectorTerm(nil)}: timestamp=2021-07-06T10:25:55.858Z

2021-07-06T10:25:55.956Z [INFO]  plugin.terraform-provider-kubernetes_v2.3.2_x5: 2021/07/06 10:25:55 [INFO] Received storage class: &v1.StorageClass{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"csi-gp2", GenerateName:"", Namespace:"", SelfLink:"", UID:"e29db48f-c27a-4c7d-a806-accf17e622d6", ResourceVersion:"3519662", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63759526795, loc:(*time.Location)(0x325c660)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"storageclass.kubernetes.io/is-default-class":"true"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kubectl-patch", Operation:"Update", APIVersion:"storage.k8s.io/v1", Time:(*v1.Time)(0xc00119e228), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00119e240)}, v1.ManagedFieldsEntry{Manager:"HashiCorp", Operation:"Update", APIVersion:"storage.k8s.io/v1", Time:(*v1.Time)(0xc00119e258), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00119e270)}, v1.ManagedFieldsEntry{Manager:"kubectl-edit", Operation:"Update", APIVersion:"storage.k8s.io/v1", Time:(*v1.Time)(0xc00119e288), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00119e2a0)}}}, Provisioner:"ebs.csi.aws.com", Parameters:map[string]string{"type":"gp2"}, ReclaimPolicy:(*v1.PersistentVolumeReclaimPolicy)(0xc0011b21e0), MountOptions:[]string(nil), AllowVolumeExpansion:(*bool)(0xc00119a28e), VolumeBindingMode:(*v1.VolumeBindingMode)(0xc0011b21f0), AllowedTopologies:[]v1.TopologySelectorTerm(nil)}: timestamp=2021-07-06T10:25:55.955Z

2021/07/06 10:25:55 [WARN] Provider "registry.terraform.io/hashicorp/kubernetes" produced an unexpected new value for module.ebs_csi_driver[0].kubernetes_storage_class.csi-gp2, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .allow_volume_expansion: was cty.True, but now cty.False

Steps to Reproduce

1. Apply a kubernetes_storage_class resource with allow_volume_expansion set to either true or false
2. Verify volume expansion setting with kubectl get sc
3. Toggle allow_volume_expansion

Expected Behavior

allow_volume_expansion setting appears as a diff in plan output, applying the configuration toggles allowVolumeExpansion according to specification.

Actual Behavior

allow_volume_expansion setting does appear as a diff in plan output, but allowVolumeExpansion seems to retain the setting it was initially provisioned with.

Explicitly tainting the resource and running apply provisions with allowVolumeExpansion set according to spec.

References

We get very similar warning in debug output as mentioned in https://discuss.hashicorp.com/t/context-around-the-log-entry-tolerating-it-because-it-is-using-the-legacy-plugin-sdk/1630.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[eddy-curv]

This also happens to me on GKE v1.18.17-gke.1901 and the following terraform versions: Terraform v0.13.7

• provider registry.terraform.io/hashicorp/kubernetes v1.13.2

regardless of the initial value of allow_volume_expansion, changing it in terraform only shows the change in plan output, but doesn't actually change the allowVolumeExpansion property in the yaml.

  # module.cloud_nodes.module.kubernetes.kubernetes_storage_class.non_expandable_ssd will be updated in-place
  ~ resource "kubernetes_storage_class" "non_expandable_ssd" {
      ~ allow_volume_expansion = false -> true
        id                     = "non-expandable-ssd"
        mount_options          = []
        parameters             = {
            "type" = "pd-ssd"
        }
        reclaim_policy         = "Delete"
        storage_provisioner    = "kubernetes.io/gce-pd"
        volume_binding_mode    = "Immediate"

        metadata {
            annotations      = {}
            generation       = 0
            labels           = {}
            name             = "non-expandable-ssd"
            resource_version = "350836511"
            self_link        = "/apis/storage.k8s.io/v1/storageclasses/non-expandable-ssd"
            uid              = "998b3a10-913f-46ec-8eeb-4bd1fc060f9e"
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.
module.cloud_nodes.module.kubernetes.kubernetes_storage_class.non_expandable_ssd: Modifying... [id=non-expandable-ssd]
module.cloud_nodes.module.kubernetes.kubernetes_storage_class.non_expandable_ssd: Modifications complete after 2s [id=non-expandable-ssd]

after running apply, the storageclasses yaml remains the same (updated timestamp doesn't change):

allowVolumeExpansion: false
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2021-07-15T13:31:08Z"
  managedFields:
  - apiVersion: storage.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:allowVolumeExpansion: {}
      f:parameters:
        .: {}
        f:type: {}
      f:provisioner: {}
      f:reclaimPolicy: {}
      f:volumeBindingMode: {}
    manager: HashiCorp
    operation: Update
    time: "2021-07-15T13:31:08Z"
  name: non-expandable-ssd
  resourceVersion: "350836511"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/non-expandable-ssd
  uid: 998b3a10-913f-46ec-8eeb-4bd1fc060f9e
parameters:
  type: pd-ssd
provisioner: kubernetes.io/gce-pd
reclaimPolicy: Delete
volumeBindingMode: Immediate
 

[dak1n1]

Thanks for this report! I was able to reproduce this. I'm seeing a perpetual diff. Our update function is not working here and needs to be fixed. It looks like create is working, so the attribute can be set at create time. As a workaround, kubectl edit is able to edit this field, but we still need to fix this on the provider side as soon as possible.

[scotthew1]

@dak1n1 is this a problem with the update function in general or specific to allow_volume_expansion? i'm seeing a similar issue trying to change reclaim_policy from Delete to Retain

[euven]

@dak1n1 is there a reason why the related pull request has not been merged?

Thanks!