openebs
LVM node can't create new PV
What steps did you take and what happened: [A clear and concise description of what the bug is, and what commands you ran.] The lvm csi runs correctly at first. But after a longtime running(maybe two months), I create a StatefulSet with lvm pvc, but the pv can't be created.
What did you expect to happen: PV can be successfully created.
The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other Pastebin is fine.)
describe pod:
kubectl get pods -n openebs
# kubectl get pod -n openebs
NAME READY STATUS RESTARTS AGE
kg-lvm-csi-lvm-localpv-controller-8487d5745c-4ln7x 5/5 Running 0 5d
kg-lvm-csi-lvm-localpv-node-54tmn 2/2 Running 0 5d
kubectl get lvmvol -A -o yaml
Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.] In the csi node log, if a pv can be successfully created, the log should be as following:
I0910 07:42:51.015586 1 volume.go:85] Got add event for Vol pvc-24626e4e-6f29-4fbc-a59b-a4f9b51a6ae9
I0910 07:42:51.015605 1 volume.go:86] lvmvolume object to be enqueued by Add handler:
I0910 07:42:51.015690 1 volume.go:55] Getting lvmvol object name:pvc-24626e4e-6f29-4fbc-a59b-a4f9b51a6ae9, ns from cache
However, in my environment, we only get the lines "Got add event for Vo" and "lvmvolume object to be enqueued by Add handler", without the third line. After I reboot the csi node pod, everything is fine again. I can create pv successfully.
So, I doubt that there is something wrong with the workqueue, LVMVolume is put into the workqueue, but in the processNextWorkItem loop, nothing can be get from the queue.
But I'm not sure with that, is there a clue ?
Environment:
- LVM Driver version: 1.5.0
- Kubernetes version (use
kubectl version):1.24.1 - Kubernetes installer & version:
- Cloud provider or hardware configuration:
- OS (e.g. from
/etc/os-release):
Hi @styshoo , Can you provide more details on the workflow..
kubectl get sc <sc-name>
kubectl get pvc <pvc-name>
kubectl get lvmvolume <pv-name> -n openebs
Do you mean after restarting lvm-node plugin lv gets created successfully?
Please share above requested info, and complete logs of both lvm pods if available, in case this issue is reproducible.
