unable to mount volume for statefulset application when the pods moved from nodes

[survivant]

new usecase problem with statefulset cstor mount : here what I did. 0 - kubectl get statefulset maria -o yaml > statefulset-maria.yaml 1 - I remove my statefulset application that was running in node test110 (that node after that stop been healthy : PUGL warning) 2 - I updated my statefulset to change the database image mariadb 1.3.20->1.3.27 3 - kubectl apply -f statefulset-maria.yaml 4 - kubernetes installed it on node test 111 this time. 5 - I kill the pods few times.. and each time the database came back running fine 6 - the node 110 came back to life. So I kill my pod, and the pod was move to test110 7 - the pod is in ContainerCreating over 30 min. (got the event : MountVolume.WaitForAttach failed for volume "pvc-4422472a-6e1c-4b03-9287-63d8ce5aa793" : rpc error: code = Internal desc = Volume still mounted on node: test-pcl110) 8 - I did a describe on this

root@test-pcl109:/tmp# kubectl get volumeattachments csi-a59a3cd375d34f5604e49c735bff447820c4d2b4cef258b9a4a1bf1a6007a806
NAME                                                                   ATTACHER               PV                                         NODE          ATTACHED   AGE
csi-a59a3cd375d34f5604e49c735bff447820c4d2b4cef258b9a4a1bf1a6007a806   cstor.csi.openebs.io   pvc-4422472a-6e1c-4b03-9287-63d8ce5aa793   test-pcl110   false      18m

root@test-pcl109:/tmp# kubectl describe volumeattachments csi-a59a3cd375d34f5604e49c735bff447820c4d2b4cef258b9a4a1bf1a6007a806
Name:         csi-a59a3cd375d34f5604e49c735bff447820c4d2b4cef258b9a4a1bf1a6007a806
Namespace:
Labels:       <none>
Annotations:  csi.alpha.kubernetes.io/node-id: test-pcl110
API Version:  storage.k8s.io/v1
Kind:         VolumeAttachment
Metadata:
  Creation Timestamp:  2020-12-09T16:10:52Z
  Finalizers:
    external-attacher/cstor-csi-openebs-io
  Managed Fields:
    API Version:  storage.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:attacher:
        f:nodeName:
        f:source:
          f:persistentVolumeName:
    Manager:      kube-controller-manager
    Operation:    Update
    Time:         2020-12-09T16:10:52Z
    API Version:  storage.k8s.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:csi.alpha.kubernetes.io/node-id:
        f:finalizers:
          .:
          v:"external-attacher/cstor-csi-openebs-io":
      f:status:
        f:attachError:
          .:
          f:message:
          f:time:
    Manager:         csi-attacher
    Operation:       Update
    Time:            2020-12-09T16:29:09Z
  Resource Version:  73111510
  Self Link:         /apis/storage.k8s.io/v1/volumeattachments/csi-a59a3cd375d34f5604e49c735bff447820c4d2b4cef258b9a4a1bf1a6007a806
  UID:               4626d5a2-0519-47f0-96be-99837947dd79
Spec:
  Attacher:   cstor.csi.openebs.io
  Node Name:  test-pcl110
  Source:
    Persistent Volume Name:  pvc-4422472a-6e1c-4b03-9287-63d8ce5aa793
Status:
  Attach Error:
    Message:  rpc error: code = Internal desc = Volume still mounted on node: test-pcl110
    Time:     2020-12-09T16:29:09Z
  Attached:   false
Events:       <none>

9 - What I do now ? 10 - What I do to be sure that it won't happen again.. I'm trying to find a way to do that automaticaly.

[mittachaitu]

This issue has been handled in 2.5.0 via 134 and there were subsequent key improvements on this issue in 2.12.0 version of OpenEBS. Please verify if it is still able to reproduce in OpenEBS 2.12.0 version.

[survivant]

I'll try

[blampe]

FWIW I'm still seeing this in 3.3.0.