Multiple Requests are coming to CSI for resizing for invalid value

[saurabhwani5]

Describe the bug

Around 2733 requests have come to CSI for resizing invalid value which is not expected Normal Resizing 4m26s (x2733 over 3d23h) external-resizer spectrumscale.csi.ibm.com External resizer is resizing volume pvc-0b6af8a8-39d2-43e1-9535-465548222926

How to Reproduce?

1. Install CSI with following images #1226

Operator : quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator@sha256:1e579401a52f4ae12eff2b0b61b83af12d37d354a80d2ee6b83b9ac6b7cc01e2
Driver : quay.io/badri_pathak/ibm-spectrum-scale-csi-driver:pvc_max_size_v1

2. Create PVC as following :

[root@ocppr1226]# cat pvc.yaml
apiVersion: v1
kind: Pod
metadata:
  name: csi-scale-fsetdemo-pod-2
  labels:
    app: nginx
spec:
  containers:
   - name: web-server
     image: nginx:1.22.0
     volumeMounts:
       - name: mypvc
         mountPath: /usr/share/nginx/html/scale
     ports:
     - containerPort: 80
  volumes:
   - name: mypvc
     `persistentVolumeClaim:`
       claimName: scale-advance-pvc-1
       readOnly: false

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: scale-advance-pvc-1
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: ibm-spectrum-scale-csi-advance

---

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: ibm-spectrum-scale-csi-advance
provisioner: spectrumscale.csi.ibm.com
parameters:
   volBackendFs: "fs0"
   version: "2"
reclaimPolicy: Delete
allowVolumeExpansion: true
[root@ocppr1226]# oc apply -f pvc.yaml
pod/csi-scale-fsetdemo-pod-2 created
persistentvolumeclaim/scale-advance-pvc-1 created
storageclass.storage.k8s.io/ibm-spectrum-scale-csi-advance created
[root@ocppr1226]# oc get pods
NAME                                                  READY   STATUS    RESTARTS   AGE
csi-scale-fsetdemo-pod-2                              1/1     Running   0          62s
ibm-spectrum-scale-csi-attacher-5d58777ffc-75c6d      1/1     Running   0          34m
ibm-spectrum-scale-csi-attacher-5d58777ffc-d4mpl      1/1     Running   0          34m
ibm-spectrum-scale-csi-cfmnq                          3/3     Running   0          34m
ibm-spectrum-scale-csi-operator-75f546c65c-q8vt2      1/1     Running   0          35m
ibm-spectrum-scale-csi-provisioner-7f698474dd-xmhjf   1/1     Running   0          34m
ibm-spectrum-scale-csi-resizer-67d499dc54-h892q       1/1     Running   0          34m
ibm-spectrum-scale-csi-snapshotter-b9b67b997-5fw2t    1/1     Running   0          34m
ibm-spectrum-scale-csi-vzvt8                          3/3     Running   0          34m
ibm-spectrum-scale-csi-z5sl5                          3/3     Running   0          34m
[root@ocppr1226]# oc get pvc
NAME                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                     VOLUMEATTRIBUTESCLASS   AGE
scale-advance-pvc-1   Bound    pvc-0b6af8a8-39d2-43e1-9535-465548222926   1Gi        RWX            ibm-spectrum-scale-csi-advance   <unset>                 63s
[root@ocppr1226]#

3. Increase PVC size to 9223372036854775807Gi:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: scale-advance-pvc-1
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 9223372036854775807Gi
  storageClassName: ibm-spectrum-scale-csi-advance
[root@ocppr1226]# oc apply -f pvc.yaml
pod/csi-scale-fsetdemo-pod-2 configured
persistentvolumeclaim/scale-advance-pvc-1 configured
storageclass.storage.k8s.io/ibm-spectrum-scale-csi-advance unchanged

4. Check PVC description after 3 days

[[email protected] ~]# oc describe pvc
Name:          scale-advance-pvc-1
Namespace:     ibm-spectrum-scale-csi
StorageClass:  ibm-spectrum-scale-csi-advance
Status:        Bound
Volume:        pvc-0b6af8a8-39d2-43e1-9535-465548222926
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: spectrumscale.csi.ibm.com
               volume.kubernetes.io/storage-provisioner: spectrumscale.csi.ibm.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1Gi
Access Modes:  RWX
VolumeMode:    Filesystem
Used By:       csi-scale-fsetdemo-pod-2
Conditions:
  Type       Status  LastProbeTime                     LastTransitionTime                Reason  Message
  ----       ------  -----------------                 ------------------                ------  -------
  Resizing   True    Mon, 01 Jan 0001 00:00:00 +0000   Sun, 06 Oct 2024 23:10:20 -0700
Events:
  Type    Reason    Age                       From                                        Message
  ----    ------    ----                      ----                                        -------
  Normal  Resizing  2m45s (x2735 over 3d23h)  external-resizer spectrumscale.csi.ibm.com  External resizer is resizing volume pvc-0b6af8a8-39d2-43e1-9535-465548222926
[[email protected] ~]#

Expected behavior

Multiple resizing request should not come to CSI for invalid value

Data Collection and Debugging

/scale-csi/D.1229
mustgather.tar.gz

[badri-pathak]

Hi @saurabhwani5 The multiple requests are expected to come on the failure, as the VolumeExpansion will reject the request having wrong volume capacity. This is expected, so we are not able to address this issue, because handled by kubernetes itself.