piraeus-operator
piraeus-operator copied to clipboard
piraeusdatastore
unable to start piraeus-op-etcd-0
I'm trying to configure piraeus on Ubuntu 20.04.3 LTS with microk8s Cluster has got 3 nodes
patryk@kubervm1:~/git/piraeus-operator/charts/piraeus/charts$ microk8s kubectl get nodes
NAME STATUS ROLES AGE VERSION
kubervm1 Ready <none> 4h51m v1.21.4-3+e5758f73ed2a04
kubervm3 Ready <none> 4h39m v1.21.4-3+e5758f73ed2a04
kubervm2 Ready <none> 4h31m v1.21.4-3+e5758f73ed2a04
I followed https://github.com/piraeusdatastore/piraeus-operator#linstor-etcd-hostpath-persistence to create hostPath for ectd and I've got PV:
patryk@kubervm1:~/git/piraeus-operator/charts/piraeus/charts$ microk8s kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
piraeus-etcd-pv-kubervm2 1Gi RWO Retain Available 4h5m
piraeus-etcd-pv-kubervm1 1Gi RWO Retain Available 4h5m
piraeus-etcd-pv-kubervm3 1Gi RWO Retain Available 4h5m
pvc-c33e8f67-2175-4793-a266-72af425cfc11 1Gi RWO Delete Bound default/datadir-piraeus-op-etcd-0 microk8s-hostpath 158m
But after installing helm chart piraeus-op with: command
sudo microk8s helm3 install piraeus-op ./charts/piraeus --set operator.satelliteSet.kernelModuleInjectionImage=quay.io/piraeusdatastore/drbd9-focal -f /home/patryk/deployment/storagepool/storagepools.yml --set csi.kubeletPath=/var/snap/microk8s/common/var/lib/kubelet --set stork.schedulerTag=v1.21.4
ETCD pod doesnt start:
patryk@kubervm1:~/git/piraeus-operator/charts/piraeus/charts$ microk8s kubectl logs piraeus-op-etcd-0
Waiting for piraeus-op-etcd-0.piraeus-op-etcd to come up
Waiting for piraeus-op-etcd-0.piraeus-op-etcd to come up
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-09-23 14:11:51.625662 I | etcdmain: etcd Version: 3.4.15
2021-09-23 14:11:51.625689 I | etcdmain: Git SHA: aa7126864
2021-09-23 14:11:51.625692 I | etcdmain: Go Version: go1.12.17
2021-09-23 14:11:51.625694 I | etcdmain: Go OS/Arch: linux/amd64
2021-09-23 14:11:51.625697 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-09-23 14:11:51.626007 I | embed: name = piraeus-op-etcd-0
2021-09-23 14:11:51.626020 I | embed: data dir = /var/run/etcd/default.etcd
2021-09-23 14:11:51.626023 I | embed: member dir = /var/run/etcd/default.etcd/member
2021-09-23 14:11:51.626026 I | embed: heartbeat = 100ms
2021-09-23 14:11:51.626032 I | embed: election = 1000ms
2021-09-23 14:11:51.626034 I | embed: snapshot count = 100000
2021-09-23 14:11:51.626041 I | embed: advertise client URLs = http://piraeus-op-etcd-0.piraeus-op-etcd:2379
2021-09-23 14:11:51.626283 C | etcdmain: cannot access data directory: mkdir /var/run/etcd/default.etcd: permission denied
It seems it claims volume with default storage class . I cant figure out how for pass PV created piraeus-etcd-pv to piraeus-op-etcd-0. Could You please give me some hint ? From manuals It seems It just should start. I've disabled AppArmor just to check. More over I think even PV is claimed with default storage class . It should be able to use hostpath so it shouldn't be permission denied. In worst case it has access to "/var/snap/microk8s/common/default-storage/default-datadir-piraeus-op-etcd-0-pvc-c33e8f67-2175-4793-a266-72af425cfc11", and this folder exists in each node ( on one i've created it manually ). Not perfect but it will work.
I tried it with one node cluster and command
microk8s helm3 install piraeus-etcd-pv ./charts/pv-hostpath --set 'nodes={kubervm1}'
and i runs , at least with default storage class.
pvc-eceb2a5b-baf5-4977-a7b4-d1336fd90710 1Gi RWO Delete Bound default/datadir-piraeus-op-etcd-0 microk8s-hostpath 10m
So the question is : How to use prepared PV by piraeus-etcd-pv ?
If you use microk8s, you can probably skip the step about pre-provisioning the PVCs, as the storage plugin and the default storage class should also just work. There seems to be another issue preventing etcd to start. Maybe we forgot to add a fsGroup: to the security context of the etcd containers :thinking:
Hard me to say now as I've rebuild all the cluster from scratch starting from one node but I think I will test it once more time with full 3-node configuration in some time so If You need any more logs I can provide.
I think I found the reason PV is claimed on node kubervm1:
root@kubervm1:/home/patryk/git/piraeus-operator# microk8s kubectl describe pv pvc-60b9cbc8-a4b9-480f-b016-40328f2ecb60 Name: pvc-60b9cbc8-a4b9-480f-b016-40328f2ecb60 Labels: <none> Annotations: hostPathProvisionerIdentity: kubervm1 pv.kubernetes.io/provisioned-by: microk8s.io/hostpath Finalizers: [kubernetes.io/pv-protection] StorageClass: microk8s-hostpath Status: Bound Claim: default/datadir-piraeus-op-etcd-0 Reclaim Policy: Delete Access Modes: RWO VolumeMode: Filesystem Capacity: 1Gi Node Affinity: <none> Message: Source: Type: HostPath (bare host directory volume) Path: /var/snap/microk8s/common/default-storage/default-datadir-piraeus-op-etcd-0-pvc-60b9cbc8-a4b9-480f-b016-40328f2ecb60 HostPathType: Events: <none>
but pod on kubervm2
root@kubervm1:/home/patryk/git/piraeus-operator# microk8s kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES busybox-kubervm2-765484f885-5844w 1/1 Running 0 68m 10.1.72.209 kubervm2 <none> <none> piraeus-op-operator-66866cf766-46825 1/1 Running 0 12m 10.1.72.210 kubervm2 <none> <none> piraeus-op-stork-scheduler-695846c974-wtknk 1/1 Running 0 12m 10.1.134.209 kubervm3 <none> <none> snapshot-controller-556cb4ccc5-hpbkh 1/1 Running 0 12m 10.1.72.211 kubervm2 <none> <none> piraeus-op-stork-65bd8569-hkzc4 1/1 Running 0 12m 10.1.134.211 kubervm3 <none> <none> piraeus-op-ns-node-rf4m6 2/2 Running 0 12m 192.168.66.2 kubervm2 <none> <none> piraeus-op-csi-controller-84b76f555-jd5hk 6/6 Running 0 12m 10.1.134.213 kubervm3 <none> <none> piraeus-op-ns-node-5v7zs 2/2 Running 0 12m 192.168.66.3 kubervm3 <none> <none> piraeus-op-csi-node-r79zb 2/3 CrashLoopBackOff 6 12m 10.1.134.212 kubervm3 <none> <none> piraeus-op-csi-node-9226k 2/3 CrashLoopBackOff 6 12m 10.1.72.213 kubervm2 <none> <none> piraeus-op-csi-node-qhpk5 2/3 CrashLoopBackOff 6 12m 10.1.25.66 kubervm1 <none> <none> piraeus-op-ns-node-rfk96 1/2 CrashLoopBackOff 6 12m 192.168.66.1 kubervm1 <none> <none> piraeus-op-cs-controller-86b4b69b9c-c9wdr 0/1 CrashLoopBackOff 5 12m 10.1.72.214 kubervm2 <none> <none> piraeus-op-ha-controller-6dff997bc7-2t7sn 0/1 CrashLoopBackOff 7 12m 10.1.134.210 kubervm3 <none> <none> piraeus-op-etcd-0 0/1 CrashLoopBackOff 7 12m 10.1.72.212 kubervm2 <none> <none>