k8s-node-termination-handler
k8s-node-termination-handler copied to clipboard
Error thrown on GKE cluster, permission issue
Hi,
I just deployed your application, created a Helm chart for it, but I get this error with your ClusterRole:
I0114 09:24:30.253282 10558 round_trippers.go:405] GET https://10.92.128.1:443/api/v1/nodes/gke-dashur-dev-dashur-dev-nodepool-2c0fca4c-xjcv 403 Forbidden in 22 milliseconds I0114 09:24:30.253309 10558 round_trippers.go:411] Response Headers: I0114 09:24:30.253314 10558 round_trippers.go:414] Content-Length: 409 I0114 09:24:30.253318 10558 round_trippers.go:414] Date: Tue, 14 Jan 2020 09:24:30 GMT I0114 09:24:30.253322 10558 round_trippers.go:414] Audit-Id: 58e9a508-c00c-4d00-a71e-7a55fbfa1e24 I0114 09:24:30.253326 10558 round_trippers.go:414] Content-Type: application/json I0114 09:24:30.253329 10558 round_trippers.go:414] X-Content-Type-Options: nosniff I0114 09:24:30.253355 10558 request.go:874] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"nodes \"gke-dashur-dev-dashur-dev-nodepool-2c0fca4c-xjcv\" is forbidden: User \"system:serviceaccount:kube-system:node-terrmination-handler\" cannot get resource \"nodes\" in API group \"\" at the cluster scope","reason":"Forbidden","details":{"name":"gke-dashur-dev-dashur-dev-nodepool-2c0fca4c-xjcv","kind":"nodes"},"code":403} I0114 09:24:30.253923 10558 taint.go:81] Failed to remove taint: nodes "gke-dashur-dev-dashur-dev-nodepool-2c0fca4c-xjcv" is forbidden: User "system:serviceaccount:kube-system:node-terrmination-handler" cannot get resource "nodes" in API group "" at the cluster scope I0114 09:24:30.253947 10558 handler.go:90] Failed to process initial node state - nodes "gke-dashur-dev-dashur-dev-nodepool-2c0fca4c-xjcv" is forbidden: User "system:serviceaccount:kube-system:node-terrmination-handler" cannot get resource "nodes" in API group "" at the cluster scope F0114 09:24:30.253957 10558 main.go:87] nodes "gke-dashur-dev-dashur-dev-nodepool-2c0fca4c-xjcv" is forbidden: User "system:serviceaccount:kube-system:node-terrmination-handler" cannot get resource "nodes" in API group "" at the cluster scope
Looks like the ClusterRole is still missing something to access the node pool. Any idea what needs to be added?
Kind regards,
Eric V.
Can you verify that https://github.com/GoogleCloudPlatform/k8s-node-termination-handler/blob/master/deploy/rbac.yaml#L35 is correctly applied on your cluster?
Hello,
Thanks for your reply, this is what's current in place on my cluster:
╰─$ (⎈ gke_dashur-dev_asia-southeast1-a_dashur-dev:kube-system) k get clusterroles node-termination-handler -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: "2020-01-14T09:13:27Z"
labels:
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/version: 1.0.0
helm.sh/chart: node-termination-handler-1.0.0
k8s-app: node-termination-handler
name: node-termination-handler
name: node-termination-handler
resourceVersion: "77819"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/node-termination-handler
uid: 262267b3-66d4-4acc-9576-8fa84929afbe
rules:
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- update
- apiGroups:
- ""
resources:
- events
verbs:
- create
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- list
- delete
and the clusterrolebinding:
╰─$ (⎈ gke_dashur-dev_asia-southeast1-a_dashur-dev:kube-system) k get clusterrolebindings node-termination-handler -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: "2020-01-14T09:13:27Z"
labels:
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/version: 1.0.0
helm.sh/chart: node-termination-handler-1.0.0
k8s-app: node-termination-handler
name: node-termination-handler
name: node-termination-handler
resourceVersion: "77822"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/node-termination-handler
uid: 91d1143b-ee48-48a4-8c24-dabb0205fed8
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: node-termination-handler
subjects:
- kind: ServiceAccount
name: node-termination-handler
namespace: kube-system
I temporarily assigned cluster-admin role to the service account to get the Pods working.
Kind regards,
Eric V.