[Large scale test]Evicte 100 pods that use storage,There is a lvmigrate stuck in SyncReplica

[FloatXD]

Evicte 100 pods that use storage,There is a lvmigrate stuck in SyncReplica

[root@k8s-master ~]# kubectl describe lvmigrate evictor-pvc-77849b03-8306-4f50-8e91-d0075acdd90f
Name:         evictor-pvc-77849b03-8306-4f50-8e91-d0075acdd90f
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  hwameistor.io/v1alpha1
Kind:         LocalVolumeMigrate
Metadata:
  Creation Timestamp:  2023-12-11T07:29:49Z
  Finalizers:
    hwameistor.io/eviction-protect
  Generation:        1
  Resource Version:  288902
  UID:               9ed451d2-d2b7-4d30-b88a-08989e252b8e
Spec:
  Abort:             false
  Migrate All Vols:  true
  Source Node:       k8s-node1
  Target Nodes Suggested:
  Volume Name:  pvc-77849b03-8306-4f50-8e91-d0075acdd90f
Status:
  Message:                  waiting for the sync job to complete: evictor-pvc-77849b03-8306-datacopy-local-storage-pvc-lvm30
  Original Replica Number:  1
  State:                    SyncReplica
  Target Node:              k8s-node2
  Volumes:
    pvc-77849b03-8306-4f50-8e91-d0075acdd90f
Events:  <none>

[root@k8s-master ~]# kubectl get pod -n hwameistor
NAME                                                       READY   STATUS             RESTARTS      AGE
hwameistor-admission-controller-d684c7b75-88n5w            1/1     Running            0             22m
hwameistor-admission-controller-d684c7b75-f8sm9            1/1     Terminating        2 (58m ago)   3d5h
hwameistor-admission-controller-d684c7b75-k95rf            1/1     Running            0             22m
hwameistor-admission-controller-d684c7b75-smb4v            1/1     Running            0             18m
hwameistor-apiserver-6f6579b5c-kkrt7                       1/1     Running            1 (61m ago)   3d5h
hwameistor-auditor-6778bb5df7-vssrg                        0/1     CrashLoopBackOff   8 (2m ago)    18m
hwameistor-exporter-75cc84967-bghls                        1/1     Running            1 (61m ago)   3d5h
hwameistor-failover-assistant-6c675d589b-rtr22             1/1     Running            1 (61m ago)   3d5h
hwameistor-local-disk-action-controller-6878df7545-nlj5r   1/1     Running            0             18m
hwameistor-local-disk-csi-controller-6f6fc5875b-jqwr9      2/2     Running            2 (61m ago)   3d5h
hwameistor-local-disk-manager-b2f9j                        2/2     Running            2 (61m ago)   3d5h
hwameistor-local-disk-manager-q4972                        2/2     Running            2 (61m ago)   3d5h
hwameistor-local-disk-manager-tr7dk                        2/2     Running            2 (61m ago)   3d5h
hwameistor-local-storage-c4f5w                             2/2     Running            2 (61m ago)   3d5h
hwameistor-local-storage-csi-controller-69dd5654dc-tj8jp   6/6     Running            6 (61m ago)   3d5h
hwameistor-local-storage-l6bs8                             2/2     Running            3 (59m ago)   3d5h
hwameistor-local-storage-vkf4s                             2/2     Running            2 (61m ago)   3d5h
hwameistor-pvc-autoresizer-648f4d7bbd-lcdc6                1/1     Running            1 (61m ago)   3d5h
hwameistor-scheduler-58cb757788-2gr7t                      1/1     Running            0             21m
hwameistor-scheduler-58cb757788-46r56                      1/1     Running            0             21m
hwameistor-scheduler-58cb757788-7tqpd                      1/1     Terminating        1 (61m ago)   3d5h
hwameistor-scheduler-58cb757788-x2v4n                      1/1     Running            0             18m
hwameistor-ui-78756c7557-9qn9d                             1/1     Running            1 (61m ago)   3d5h
hwameistor-volume-evictor-569cbf4bcd-w8xmh                 1/1     Running            0             18m
hwameistor-volume-evictor-569cbf4bcd-ztm6b                 1/1     Terminating        1 (61m ago)   3d5h

[root@k8s-master ~]# kubectl get lvr |grep node1
pvc-77849b03-8306-4f50-8e91-d0075acdd90f-kkch5t   20971520   k8s-node1   Ready   true     /dev/LocalStorage_PoolHDD/pvc-77849b03-8306-4f50-8e91-d0075acdd90f   52m
[root@k8s-master ~]# kubectl get lv |grep d0075acdd90f
pvc-77849b03-8306-4f50-8e91-d0075acdd90f   LocalStorage_PoolHDD   2          20971520   1134592   Ready               52m
[root@k8s-master ~]# kubectl get pvc |grep pvc-77849b03-8306-4f50-8e91-d0075acdd90f
local-storage-pvc-lvm30   Bound    pvc-77849b03-8306-4f50-8e91-d0075acdd90f   20Mi       RWO            local-storage-hdd-lvm   54m

[buffalo1024]

can you have a look at the job of the lvmigrate? you can have a try to delete the job and the same name job should be created automatically soon.

[FloatXD]

can you have a look at the job of the lvmigrate? you can have a try to delete the job and the same name job should be created automatically soon.

After deleting the job, migrate proceeds smoothly. But why does this happen?

[github-actions[bot]]

This issue has been marked as stale because it has been open for 90 days with no activity. This thread will be automatically closed in 30 days if no further activity occurs.