k3s - HTTP status 401 Unauthorized

[NicklasWallgren]

trafficstars

What did you do? We've deployed kube-prometheus to our k3s cluster using the Quickstart Guide.

Did you expect to see some different? All targets healthy and not reported as down in prometheus. The problem is probably related to RBAC.

Prometheus Targets

monitoring/kube-state-metrics/0 (0/1 up)

https://10.42.0.11:8443/metrics
server returned HTTP status 401 Unauthorized

monitoring/kube-state-metrics/1 (0/1 up)

server returned HTTP status 401 Unauthorized
https://10.42.0.11:9443/metrics

monitoring/node-exporter/0 (0/1 up)

https://159.65.aaa.bbb:9100/metrics
server returned HTTP status 401 Unauthorized

Environment K3s 0.8.1 K8s v1.14.6 Kube-prometheus 0.2

[NicklasWallgren]

Log output.

kube-rbac-proxy-main

E0921 12:14:01.518665       1 webhook.go:106] Failed to make webhook authenticator request: the server could not find the requested resource
E0921 12:14:01.518704       1 proxy.go:67] Unable to authenticate the request due to an error: the server could not find the requested resource

kube-rbac-proxy-self

E0921 12:15:19.846455       1 webhook.go:106] Failed to make webhook authenticator request: the server could not find the requested resource
E0921 12:15:19.846494       1 proxy.go:67] Unable to authenticate the request due to an error: the server could not find the requested resource

[NicklasWallgren]

Found the cause of the issue.

I found my issue with k8s audit, kube-rbac-proxy use authentication.k8s.io/v1beta1 to authenticate the token, but k3s removed v1beta1 in favor to v1.

https://github.com/carlosedp/cluster-monitoring/issues/13

[paulfantom]

Cannot be easily fixed until kubernetes api server migrates to v1 in https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/authentication/authenticatorfactory/delegating.go#L36

[brancz]

Maybe I'm totally misunderstanding, but doesn't that just mean we need to update kube-rbac-proxy to use the v1 API?

[paulfantom]

Yes, that's exactly what needs to be done and I started doing that. However, there is a transient dependency on v1beta1 coming from https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/authentication/authenticatorfactory/delegating.go#L36

[brancz]

I see. Maybe worth asking in kubernetes slack if this is a left over, and just needs to be done.

[paulfantom]

Asked. This change is not planned yet.

[AshtonKem]

Just checking in on this. k3s no longer supports rbac v1beta1.

[jodevsa]

any updates? somehow this is causing kube-rbac-proxy to throttle

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the last 60 days. Thank you for your contributions.

[github-actions[bot]]

This issue was closed because it has not had any activity in the last 120 days. Please reopen if you feel this is still valid.