kube-arangodb
kube-arangodb copied to clipboard
arango-storage Cannot set leader role on Pod
I have a cluster with one master node and one worker node. When I try to create arango-storage the status gets stuck on CrashLoopBackOff (see get pods output below). Also, from the logs output it seems the problem has to do with setting one of the pods with the leader role.
Am I doing something wrong here?
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:21:50Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.10", GitCommit:"098570796b32895c38a9a1c9286425fb1ececa18", GitTreeState:"clean", BuildDate:"2018-08-02T17:11:51Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default arango-deployment-operator-cfbbcd47d-6xt7d 1/1 Running 0 23h
default arango-deployment-operator-cfbbcd47d-mpn8q 1/1 Running 0 23h
default arango-storage-operator-5fb6c68c9d-hwnls 0/1 CrashLoopBackOff 5 5m
default arango-storage-operator-5fb6c68c9d-pb6td 0/1 CrashLoopBackOff 5 5m
kube-system calico-node-gzzbs 2/2 Running 0 23h
kube-system calico-node-qczfg 2/2 Running 0 23h
kube-system etcd-kubeadm-master 1/1 Running 0 23h
kube-system kube-apiserver-kubeadm-master 1/1 Running 0 23h
kube-system kube-controller-manager-kubeadm-master 1/1 Running 0 23h
kube-system kube-dns-6f4fd4bdf-jl6nz 3/3 Running 0 23h
kube-system kube-proxy-4xtqz 1/1 Running 0 23h
kube-system kube-proxy-j8dhm 1/1 Running 0 23h
kube-system kube-scheduler-kubeadm-master 1/1 Running 0 23h
$ kubectl logs arango-storage-operator-5fb6c68c9d-hwnls
2018-08-17T19:07:50Z |INFO| Starting arangodb-operator, version 0.2.2 build 0ebc340 operator-id=hwnls pod-name=arango-storage-operator-5fb6c68c9d-hwnls pod-namespace=default
2018-08-17T19:07:50Z |INFO| attempting to acquire leader lease default/arango-storage-operator... component=glog operator-id=hwnls
2018-08-17T19:07:50Z |INFO| New leader detected component=operator identity=arango-storage-operator-5fb6c68c9d-pb6td-arango-storage-operator-5fb6c68c9d-pb6td lock-name=arango-storage-operator operator-id=hwnls
2018-08-17T19:08:08Z |INFO| successfully acquired lease default/arango-storage-operator component=glog operator-id=hwnls
2018-08-17T19:08:08Z |INFO| New leader detected component=operator identity=arango-storage-operator-5fb6c68c9d-hwnls-arango-storage-operator-5fb6c68c9d-hwnls lock-name=arango-storage-operator operator-id=hwnls
2018-08-17T19:08:08Z |INFO| Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"default", Name:"arango-storage-operator", UID:"4f6d7a5d-a250-11e8-9c1a-fa163e0c1541", APIVersion:"v1", ResourceVersion:"149084", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' arango-storage-operator-5fb6c68c9d-hwnls-arango-storage-operator-5fb6c68c9d-hwnls became leader operator-id=hwnls
2018-08-17T19:08:08Z |INFO| Event(v1.ObjectReference{Kind:"Deployment", Namespace:"default", Name:"arango-storage-operator", UID:"4cdc1b2b-a250-11e8-9c1a-fa163e0c1541", APIVersion:"apps", ResourceVersion:"148987", FieldPath:""}): type: 'Normal' reason: 'Leader Election Won' Pod arango-storage-operator-5fb6c68c9d-hwnls is running as leader operator-id=hwnls
2018-08-17T19:08:08Z |ERRO| Failed to update Pod wrt 'role' label component=operator error="pods \"arango-storage-operator-5fb6c68c9d-hwnls\" is forbidden: User \"system:serviceaccount:default:arango-storage-operator\" cannot update pods in the namespace \"default\"" lock-name=arango-storage-operator operator-id=hwnls pod-name=arango-storage-operator-5fb6c68c9d-hwnls
2018-08-17T19:08:08Z |ERRO| Cannot set leader role on Pod. Terminating process component=operator lock-name=arango-storage-operator operator-id=hwnls