ceph-csi
ceph-csi copied to clipboard
ceph
rbd: add volume locks for reclaimspace operations
Describe what this PR does
This commit adds VolumeLocks on reclaimspace operations to prevent multiple process executing rbd sparsify/fstrim on same volume.
Related issues
Closes: #4637
Checklist:
- [x] Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
- [x] Reviewed the developer guide on Submitting a Pull Request
- [x] Pending release notes updated with breaking and/or notable changes for the next major release.
- [x] Documentation has been updated, if necessary.
- [x] Unit tests have been added, if necessary.
- [x] Integration tests have been added, if necessary.
Show available bot commands
These commands are normally not required, but in case of issues, leave any of the following bot commands in an otherwise empty comment in this PR:
/retest ci/centos/<job-name>: retest the<job-name>after unrelated failure (please report the failure too!)
Created a PVC of 1Gi and wrote dummy data of 500M and deleted it.
root@rbd-pod-test:/# dd if=/dev/zero of=/var/lib/www/html/dummy-file bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 1.6972 s, 309 MB/s
root@rbd-pod-test:/# sync
root@rbd-pod-test:/# rm /var/lib/www/html/dummy-file
root@rbd-pod-test:/# ls -l /var/lib/www/html/
total 16
drwx------ 2 root root 16384 May 29 07:23 lost+found
before ReclaimSpace operation -
[root@c1 /]# rbd du -p replicapool
warning: fast-diff map is not enabled for csi-vol-58a49f13-ff2e-4415-b1a8-21ec59d95017. operation may be slow.
NAME PROVISIONED USED
csi-vol-58a49f13-ff2e-4415-b1a8-21ec59d95017 1 GiB 528 MiB
created ReclaimSpaceJob for the PVC
pm@dhcp53-176:~$ cat <<EOF | k create -f -
> apiVersion: csiaddons.openshift.io/v1alpha1
kind: ReclaimSpaceJob
metadata:
name: sample-1
spec:
target:
persistentVolumeClaim: rbd-pvc-test
backOffLimit: 10
retryDeadlineSeconds: 900
timeout: 600
> EOF
reclaimspacejob.csiaddons.openshift.io/sample-1 created
/NodeReclaimSpace - fstrim operation succeed
I0529 07:29:28.134875 2566810 utils.go:198] ID: 23 GRPC call: /reclaimspace.ReclaimSpaceNode/NodeReclaimSpace
I0529 07:29:28.138386 2566810 utils.go:199] ID: 23 GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.rbd.csi.ceph.com/cff6345ad1aff4cb25770de65b8012a934530bfd9d29bc77c557332df5607126/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_id":"0001-0009-rook-ceph-0000000000000008-58a49f13-ff2e-4415-b1a8-21ec59d95017"}
I0529 07:29:28.439310 2566810 cephcmds.go:105] ID: 23 command succeeded: fstrim [/var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.rbd.csi.ceph.com/cff6345ad1aff4cb25770de65b8012a934530bfd9d29bc77c557332df5607126/globalmount/0001-0009-rook-ceph-0000000000000008-58a49f13-ff2e-4415-b1a8-21ec59d95017]
[root@c1 /]# rbd du -p replicapool
warning: fast-diff map is not enabled for csi-vol-58a49f13-ff2e-4415-b1a8-21ec59d95017. operation may be slow.
NAME PROVISIONED USED
csi-vol-58a49f13-ff2e-4415-b1a8-21ec59d95017 1 GiB 304 MiB
/ControllerReclaimSpace - sparsify command succeed
I0529 07:29:28.589924 1 utils.go:198] ID: 26 GRPC call: /reclaimspace.ReclaimSpaceController/ControllerReclaimSpace
I0529 07:29:28.595724 1 utils.go:199] ID: 26 GRPC request: {"parameters":{"clusterID":"rook-ceph","encryptionKMSID":"azure-test","imageFeatures":"layering","imageFormat":"2","imageName":"csi-vol-58a49f13-ff2e-4415-b1a8-21ec59d95017","journalPool":"replicapool","pool":"replicapool","storage.kubernetes.io/csiProvisionerIdentity":"1716966822191-8666-rook-ceph.rbd.csi.ceph.com"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000008-58a49f13-ff2e-4415-b1a8-21ec59d95017"}
I0529 07:29:28.690999 1 omap.go:89] ID: 26 got omap values: (pool="replicapool", namespace="", name="csi.volume.58a49f13-ff2e-4415-b1a8-21ec59d95017"): map[csi.imageid:73c2aff800e79 csi.imagename:csi-vol-58a49f13-ff2e-4415-b1a8-21ec59d95017 csi.volname:pvc-ffd7ee84-6ae9-47fc-8eb0-32d9c255b92b csi.volume.owner:reclaim-ns]
I0529 07:29:28.761741 1 reclaimspace.go:77] ID: 26 volume with ID "0001-0009-rook-ceph-0000000000000008-58a49f13-ff2e-4415-b1a8-21ec59d95017" is in use, skipping sparsify operation
I0529 07:29:28.762193 1 utils.go:205] ID: 26 GRPC response: {}
I0529 07:31:26.161922 1 utils.go:198] ID: 27 GRPC call: /reclaimspace.ReclaimSpaceController/ControllerReclaimSpace
I0529 07:31:26.163979 1 utils.go:199] ID: 27 GRPC request: {"parameters":{"clusterID":"rook-ceph","encryptionKMSID":"azure-test","imageFeatures":"layering","imageFormat":"2","imageName":"csi-vol-58a49f13-ff2e-4415-b1a8-21ec59d95017","journalPool":"replicapool","pool":"replicapool","storage.kubernetes.io/csiProvisionerIdentity":"1716966822191-8666-rook-ceph.rbd.csi.ceph.com"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000008-58a49f13-ff2e-4415-b1a8-21ec59d95017"}
I0529 07:31:26.171744 1 omap.go:89] ID: 27 got omap values: (pool="replicapool", namespace="", name="csi.volume.58a49f13-ff2e-4415-b1a8-21ec59d95017"): map[csi.imageid:73c2aff800e79 csi.imagename:csi-vol-58a49f13-ff2e-4415-b1a8-21ec59d95017 csi.volname:pvc-ffd7ee84-6ae9-47fc-8eb0-32d9c255b92b csi.volume.owner:reclaim-ns]
I0529 07:31:26.697576 1 utils.go:205] ID: 27 GRPC response: {}
[root@c1 /]# rbd du -p replicapool
warning: fast-diff map is not enabled for csi-vol-58a49f13-ff2e-4415-b1a8-21ec59d95017. operation may be slow.
NAME PROVISIONED USED
csi-vol-58a49f13-ff2e-4415-b1a8-21ec59d95017 1 GiB 28 MiB
@Madhu-1, will this be enough for the test?
