Can't resue resources in DELETING state

[kvaps]

I already wrote about resources which are stuck in DELETING state https://github.com/LINBIT/linstor-server/issues/112 until all the nodes containing all the replicas become online. This issue about the another problem which is made out of this.

Here is quite usual situation when some node is failed, the Kubernetes or OpenNebula is moving the pod to another node.

After a while it can be moved again and again, by user or controller, it does not matter.

In the end we're getting the next situation:

The resource can be run once on each node, because after the migration old replica is stuck on DELETING state and can't be reused.

Steps to reproduce:

You have 5 nodes: node1, node2, node3, node4, node5

• Create diskful resource on node1 and node2
• Create diskless resource on node3
• Shutdown the node3
• Run VM on node4
• Migrate VM on node5
• Migrate VM back on node4

[kvaps]

Hi, just another case, pod is stuck on init state because of that:

# kubectl get pod  wes1-mx7-0
NAME         READY   STATUS     RESTARTS   AGE
wes1-mx7-0   0/7     Init:0/1   0          3m13s

# kubectl describe pod  wes1-mx7-0 
...
Events:
  Type     Reason       Age                 From               Message
  ----     ------       ----                ----               -------
  Normal   Scheduled    3m17s               default-scheduler  Successfully assigned hosting/wes1-mx7-0 to m6c37
  Warning  FailedMount  74s                 kubelet, m6c37     Unable to attach or mount volumes: unmounted volumes=[mx7-vol-data-mail], unattached volumes=[mx7-vol-data-mail cgroup fuse mx7-vol-shared mx7-sys run mx7-vol-mxbackup mx7-config-vol default-token-jt2jk]: timed out waiting for the condition
  Warning  FailedMount  54s (x9 over 3m2s)  kubelet, m6c37     MountVolume.WaitForAttach failed for volume "pvc-b1d38fcb-ee4c-4659-8973-29fdcdadb7e0" : volume attachment is being deleted

# linstor r l -r pvc-b1d38fcb-ee4c-4659-8973-29fdcdadb7e0
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName                             ┊ Node   ┊ Port  ┊ Usage  ┊ Conns ┊    State ┊ CreatedOn           ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ pvc-b1d38fcb-ee4c-4659-8973-29fdcdadb7e0 ┊ m16c38 ┊ 55184 ┊ Unused ┊ Ok    ┊ Diskless ┊ 2021-02-16 11:50:47 ┊
┊ pvc-b1d38fcb-ee4c-4659-8973-29fdcdadb7e0 ┊ m5c2   ┊ 55184 ┊ Unused ┊ Ok    ┊ UpToDate ┊                     ┊
┊ pvc-b1d38fcb-ee4c-4659-8973-29fdcdadb7e0 ┊ m5c9   ┊ 55184 ┊ Unused ┊ Ok    ┊ UpToDate ┊                     ┊
┊ pvc-b1d38fcb-ee4c-4659-8973-29fdcdadb7e0 ┊ m6c37  ┊ 55184 ┊        ┊ Ok    ┊ DELETING ┊ 2021-02-24 10:39:08 ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯