jiva-operator
jiva-operator copied to clipboard
Published
20 hours ago •
openebs-archive
openebs-archive
openebs-jiva-csi-node pod gets in CrashLoopBackOff
Hi all,
I can't get my jiva volumes working, its openebs-jiva-csi-node pod gets stuck in a CrashLoopBackOff.
What steps did you take and what happened: [A clear and concise description of what the bug is, and what commands you ran.)
- Clean up old openebs attempts according to the uninstall guide: https://openebs.io/docs/user-guides/uninstall
- Install jiva as per the quickstart guide: https://github.com/openebs/jiva-operator/blob/develop/docs/quickstart.md
kubectl apply -f https://openebs.github.io/charts/hostpath-operator.yaml
kubectl apply -f https://openebs.github.io/charts/jiva-operator.yaml
- Edit the openebs-hostpath SC to have a different BasePath (snipped config)
metadata:
annotations:
cas.openebs.io/config: |
- name: StorageType
value: "hostpath"
- name: BasePath
value: "/local-fs/virtuals/kubernetes/openebs"
- Check status of the jiva components (as per quickguide)
$ kubectl get -n openebs all
NAME READY STATUS RESTARTS AGE
pod/jiva-operator-57b879cfc8-hxnhc 1/1 Running 0 10m
pod/openebs-jiva-csi-controller-0 5/5 Running 0 10m
pod/openebs-jiva-csi-node-bd96p 2/3 CrashLoopBackOff 6 (4m48s ago) 10m
pod/openebs-localpv-provisioner-778d48fff8-9wmbr 1/1 Running 0 14m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/openebs-jiva-csi-node 1 1 0 1 0 <none> 10m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/jiva-operator 1/1 1 1 10m
deployment.apps/openebs-localpv-provisioner 1/1 1 1 14m
NAME DESIRED CURRENT READY AGE
replicaset.apps/jiva-operator-57b879cfc8 1 1 1 10m
replicaset.apps/openebs-localpv-provisioner-778d48fff8 1 1 1 14m
NAME READY AGE
statefulset.apps/openebs-jiva-csi-controller 1/1 10m
What did you expect to happen: Working Jiva
The output of the following commands will help us better understand what's going on: (Pasting long output into a GitHub gist or other pastebin is fine.)
-
kubectl describe -n openebs pod openebs-jiva-node-bd96p
Name: openebs-jiva-csi-node-bd96p
Namespace: openebs
Priority: 900001000
Priority Class Name: openebs-jiva-csi-node-critical
Node: rohan2013/192.168.1.2
Start Time: Mon, 13 Jun 2022 13:22:02 +0200
Labels: app=openebs-jiva-csi-node
controller-revision-hash=6799874754
name=openebs-jiva-csi-node
openebs.io/component-name=openebs-jiva-csi-node
openebs.io/version=3.2.0
pod-template-generation=1
role=openebs-jiva-csi
Annotations: <none>
Status: Running
IP: 192.168.1.2
IPs:
IP: 192.168.1.2
Controlled By: DaemonSet/openebs-jiva-csi-node
Containers:
csi-node-driver-registrar:
Container ID: containerd://e08df422db1a7e4e94e033a10f71b74cf53fe83d6ad5e13eedbecf42891d11e0
Image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0
Image ID: k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405
Port: <none>
Host Port: <none>
Args:
--v=5
--csi-address=$(ADDRESS)
--kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
State: Running
Started: Mon, 13 Jun 2022 13:22:03 +0200
Ready: True
Restart Count: 0
Environment:
ADDRESS: /plugin/csi.sock
DRIVER_REG_SOCK_PATH: /var/lib/kubelet/plugins/jiva.csi.openebs.io/csi.sock
KUBE_NODE_NAME: (v1:spec.nodeName)
NODE_DRIVER: openebs-jiva-csi
Mounts:
/plugin from plugin-dir (rw)
/registration from registration-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9dzsn (ro)
jiva-csi-plugin:
Container ID: containerd://bba1da5f70845cd77697a2e8f2f7523b589a298fd9de0ed904c2e5d8c803a036
Image: openebs/jiva-csi:3.2.0
Image ID: docker.io/openebs/jiva-csi@sha256:506a4d9ca03a956fe28e5fa8d3bc2960e69accd128e5f93c6024f7cfe0650151
Port: <none>
Host Port: <none>
Args:
--name=jiva.csi.openebs.io
--nodeid=$(OPENEBS_NODE_ID)
--endpoint=$(OPENEBS_CSI_ENDPOINT)
--plugin=$(OPENEBS_NODE_DRIVER)
--retrycount=20
--metricsBindAddress=:9505
State: Running
Started: Mon, 13 Jun 2022 13:22:03 +0200
Ready: True
Restart Count: 0
Environment:
OPENEBS_NODE_ID: (v1:spec.nodeName)
OPENEBS_CSI_ENDPOINT: unix:///plugin/csi.sock
OPENEBS_NODE_DRIVER: node
OPENEBS_NAMESPACE: openebs (v1:metadata.namespace)
REMOUNT: True
Mounts:
/dev from device-dir (rw)
/host from host-root (rw)
/plugin from plugin-dir (rw)
/sbin/iscsiadm from chroot-iscsiadm (rw,path="iscsiadm")
/var/lib/kubelet/ from pods-mount-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9dzsn (ro)
liveness-probe:
Container ID: containerd://098a2ff188207d6b34f10a3524d1024762e46d86f7409e6e17c6b08e9b76b6d0
Image: k8s.gcr.io/sig-storage/livenessprobe:v2.3.0
Image ID: k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d
Port: <none>
Host Port: <none>
Args:
--csi-address=/plugin/csi.sock
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Mon, 13 Jun 2022 13:33:04 +0200
Finished: Mon, 13 Jun 2022 13:33:04 +0200
Ready: False
Restart Count: 7
Environment: <none>
Mounts:
/plugin from plugin-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9dzsn (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
device-dir:
Type: HostPath (bare host directory volume)
Path: /dev
HostPathType: Directory
registration-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins_registry/
HostPathType: DirectoryOrCreate
plugin-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins/jiva.csi.openebs.io/
HostPathType: DirectoryOrCreate
pods-mount-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/
HostPathType: Directory
chroot-iscsiadm:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: openebs-jiva-csi-iscsiadm
Optional: false
host-root:
Type: HostPath (bare host directory volume)
Path: /
HostPathType: Directory
kube-api-access-9dzsn:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 12m default-scheduler Successfully assigned openebs/openebs-jiva-csi-node-bd96p to rohan2013
Normal Pulled 12m kubelet Container image "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0" already present on machine
Normal Created 12m kubelet Created container csi-node-driver-registrar
Normal Started 12m kubelet Started container csi-node-driver-registrar
Normal Pulled 12m kubelet Container image "openebs/jiva-csi:3.2.0" already present on machine
Normal Created 12m kubelet Created container jiva-csi-plugin
Normal Started 12m kubelet Started container jiva-csi-plugin
Normal Pulled 11m (x4 over 12m) kubelet Container image "k8s.gcr.io/sig-storage/livenessprobe:v2.3.0" already present on machine
Normal Created 11m (x4 over 12m) kubelet Created container liveness-probe
Normal Started 11m (x4 over 12m) kubelet Started container liveness-probe
Warning BackOff 2m34s (x48 over 12m) kubelet Back-off restarting failed container
As you can see, the liveness-probe is failing.
-
kubectl logs -n openebs openebs-jiva-csi-node-bd96p -c liveness-probe
I0613 11:33:04.652474 1 main.go:149] calling CSI driver to discover driver name
I0613 11:33:04.653452 1 main.go:155] CSI driver name: "jiva.csi.openebs.io"
I0613 11:33:04.653466 1 main.go:183] ServeMux listening at "0.0.0.0:9808"
F0613 11:33:04.653653 1 main.go:186] failed to start http server with error: listen tcp 0.0.0.0:9808: bind: address already in use
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0xc000102001, 0xc0003fa000, 0x89, 0xaa)
/workspace/vendor/k8s.io/klog/v2/klog.go:1021 +0xb9
k8s.io/klog/v2.(*loggingT).output(0xe78700, 0xc000000003, 0x0, 0x0, 0xc0000d6f50, 0xbef6a8, 0x7, 0xba, 0x0)
/workspace/vendor/k8s.io/klog/v2/klog.go:970 +0x191
k8s.io/klog/v2.(*loggingT).printf(0xe78700, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0xa708f9, 0x2a, 0xc00045cd50, 0x1, ...)
/workspace/vendor/k8s.io/klog/v2/klog.go:751 +0x191
k8s.io/klog/v2.Fatalf(...)
/workspace/vendor/k8s.io/klog/v2/klog.go:1509
main.main()
/workspace/cmd/livenessprobe/main.go:186 +0x86c
goroutine 18 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0xe78700)
/workspace/vendor/k8s.io/klog/v2/klog.go:1164 +0x8b
created by k8s.io/klog/v2.init.0
/workspace/vendor/k8s.io/klog/v2/klog.go:418 +0xdf
Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]
Environment:
- Jiva version: 3.2.0
- OpenEBS version: 3.2.0
- Kubernetes version (use
kubectl version):
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:26:19Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:18:48Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}
- Kubernetes installer & version: kubeadm
kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:24:38Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}
- Cloud provider or hardware configuration: On prem
- OS (e.g. from
/etc/os-release): Debian bullseye
Any advice or help is much appreciated :)
