kube-arangodb icon indicating copy to clipboard operation
kube-arangodb copied to clipboard

arango-storage Cannot set leader role on Pod

Open awlayton opened this issue 7 years ago • 0 comments

I have a cluster with one master node and one worker node. When I try to create arango-storage the status gets stuck on CrashLoopBackOff (see get pods output below). Also, from the logs output it seems the problem has to do with setting one of the pods with the leader role.

Am I doing something wrong here?

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:21:50Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.10", GitCommit:"098570796b32895c38a9a1c9286425fb1ececa18", GitTreeState:"clean", BuildDate:"2018-08-02T17:11:51Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                         READY     STATUS             RESTARTS   AGE
default       arango-deployment-operator-cfbbcd47d-6xt7d   1/1       Running            0          23h
default       arango-deployment-operator-cfbbcd47d-mpn8q   1/1       Running            0          23h
default       arango-storage-operator-5fb6c68c9d-hwnls     0/1       CrashLoopBackOff   5          5m
default       arango-storage-operator-5fb6c68c9d-pb6td     0/1       CrashLoopBackOff   5          5m
kube-system   calico-node-gzzbs                            2/2       Running            0          23h
kube-system   calico-node-qczfg                            2/2       Running            0          23h
kube-system   etcd-kubeadm-master                          1/1       Running            0          23h
kube-system   kube-apiserver-kubeadm-master                1/1       Running            0          23h
kube-system   kube-controller-manager-kubeadm-master       1/1       Running            0          23h
kube-system   kube-dns-6f4fd4bdf-jl6nz                     3/3       Running            0          23h
kube-system   kube-proxy-4xtqz                             1/1       Running            0          23h
kube-system   kube-proxy-j8dhm                             1/1       Running            0          23h
kube-system   kube-scheduler-kubeadm-master                1/1       Running            0          23h
$ kubectl logs arango-storage-operator-5fb6c68c9d-hwnls
2018-08-17T19:07:50Z |INFO| Starting arangodb-operator, version 0.2.2 build 0ebc340 operator-id=hwnls pod-name=arango-storage-operator-5fb6c68c9d-hwnls pod-namespace=default
2018-08-17T19:07:50Z |INFO| attempting to acquire leader lease  default/arango-storage-operator... component=glog operator-id=hwnls
2018-08-17T19:07:50Z |INFO| New leader detected component=operator identity=arango-storage-operator-5fb6c68c9d-pb6td-arango-storage-operator-5fb6c68c9d-pb6td lock-name=arango-storage-operator operator-id=hwnls
2018-08-17T19:08:08Z |INFO| successfully acquired lease default/arango-storage-operator component=glog operator-id=hwnls
2018-08-17T19:08:08Z |INFO| New leader detected component=operator identity=arango-storage-operator-5fb6c68c9d-hwnls-arango-storage-operator-5fb6c68c9d-hwnls lock-name=arango-storage-operator operator-id=hwnls
2018-08-17T19:08:08Z |INFO| Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"default", Name:"arango-storage-operator", UID:"4f6d7a5d-a250-11e8-9c1a-fa163e0c1541", APIVersion:"v1", ResourceVersion:"149084", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' arango-storage-operator-5fb6c68c9d-hwnls-arango-storage-operator-5fb6c68c9d-hwnls became leader operator-id=hwnls
2018-08-17T19:08:08Z |INFO| Event(v1.ObjectReference{Kind:"Deployment", Namespace:"default", Name:"arango-storage-operator", UID:"4cdc1b2b-a250-11e8-9c1a-fa163e0c1541", APIVersion:"apps", ResourceVersion:"148987", FieldPath:""}): type: 'Normal' reason: 'Leader Election Won' Pod arango-storage-operator-5fb6c68c9d-hwnls is running as leader operator-id=hwnls
2018-08-17T19:08:08Z |ERRO| Failed to update Pod wrt 'role' label component=operator error="pods \"arango-storage-operator-5fb6c68c9d-hwnls\" is forbidden: User \"system:serviceaccount:default:arango-storage-operator\" cannot update pods in the namespace \"default\"" lock-name=arango-storage-operator operator-id=hwnls pod-name=arango-storage-operator-5fb6c68c9d-hwnls
2018-08-17T19:08:08Z |ERRO| Cannot set leader role on Pod. Terminating process component=operator lock-name=arango-storage-operator operator-id=hwnls

awlayton avatar Aug 17 '18 19:08 awlayton