Pods are recreated because of Modification in CSO

[saurabhwani5]

Describe the bug

After installing CSI 2.9.0(dev) while running the CSI Automation after sometime in few cases one error message is shown as

2023-03-07T14:49:38.626Z        ERROR   csiscaleoperator_controller.Reconcile   Reconciler Client.Update() failed       {"error": "Operation cannot be fulfilled on csiscaleoperators.csi.ibm.com \"ibm-spectrum-scale-csi\": the object has been modified; please apply your changes to the latest version and try again"}
2023-03-07T14:49:38.660Z        ERROR   controller.csiscaleoperator     Reconciler error        {"reconciler group": "csi.ibm.com", "reconciler kind": "CSIScaleOperator", "name": "ibm-spectrum-scale-csi", "namespace": "ibm-spectrum-scale-csi-driver", "error": "Operation cannot be fulfilled on csiscaleoperators.csi.ibm.com \"ibm-spectrum-scale-csi\": the object has been modified; please apply your changes to the latest version and try again"}

How to Reproduce?

1. Install CSI 2.9.0

[root@power-csi-master tools]# oc get pods
NAME                                                  READY   STATUS    RESTARTS      AGE
ibm-spectrum-scale-csi-attacher-5885c459cb-dt2dq      1/1     Running   0             46h
ibm-spectrum-scale-csi-attacher-5885c459cb-mrxpk      1/1     Running   0             46h
ibm-spectrum-scale-csi-fw45h                          3/3     Running   0             46h
ibm-spectrum-scale-csi-operator-6dc7dd6655-x7pnf      1/1     Running   2 (46h ago)   4d21h
ibm-spectrum-scale-csi-provisioner-6b69d67cff-k55ls   1/1     Running   1 (46h ago)   46h
ibm-spectrum-scale-csi-resizer-6cc9f547d-j8mvc        1/1     Running   1 (36h ago)   46h
ibm-spectrum-scale-csi-snapshotter-6cf9b7d69b-zfgx5   1/1     Running   1 (46h ago)   46h
ibm-spectrum-scale-csi-zbd7v                          3/3     Running   0             46h

[root@power-csi-master tools]# oc get cso
NAME                     VERSION   SUCCESS
ibm-spectrum-scale-csi   2.9.0     True
[root@power-csi-master tools]# oc describe pods | grep quay
    Image:         quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver:v2.9.0-020323
    Image ID:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:48716911136ff87cd1b107aa3bfcc09b09dfb744afa82ee6e2a84ac39b7913ee
    Image:         quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator:v2.9.0-020323
    Image ID:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator@sha256:3c280b1640cac6d96d9ceefab035d7669ea91cc072f4a7debe8949714725f1f7
      CSI_DRIVER_IMAGE:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver:v2.9.0-020323
    Image:         quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver:v2.9.0-020323
    Image ID:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:48716911136ff87cd1b107aa3bfcc09b09dfb744afa82ee6e2a84ac39b7913ee

2. While running full automation found that few test cases are failing (attaching in report) with that adding logs

Expected behavior

Pod Should not restart here

[saurabhwani5]

Uploaded logs in D.948

[deeghuge]

@saurabhwani5 please do keep watch on this one. If recreated easily then we can have a look.

[deeghuge]

@saurabhwani5 please check if this is valid with latest code