metrics-server
metrics-server copied to clipboard
kubernetes-sigs
default components.yaml for 1.24+ k8s clusters lacks ServiceAccount token/secret creation
What happened: After the installation with the default components.yaml scenario the metrics-server is unable to scrape the kubelet metrics.
What you expected to happen: After the installation with the default components.yaml scenario the metrics-server is able to scrape the kubelet metrics.
Anything else we need to know?: Since 1.24+ version the token/secret for the ServiceAccount is no longer automatically created, thus metrics-server is getting loads of the errors when trying to scrape the kubelet API on port 10250:
I0325 10:23:07.712822 1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
E0325 10:23:15.451208 1 scraper.go:140] "Failed to scrape node" err="request failed, status: \"403 Forbidden\"" node="k8s-ctl3.foo.bar"
E0325 10:23:15.457767 1 scraper.go:140] "Failed to scrape node" err="request failed, status: \"403 Forbidden\"" node="k8s-wrk1.foo.bar"
E0325 10:23:15.470400 1 scraper.go:140] "Failed to scrape node" err="request failed, status: \"403 Forbidden\"" node="k8s-wrk3.foo.bar"
E0325 10:23:15.480009 1 scraper.go:140] "Failed to scrape node" err="request failed, status: \"403 Forbidden\"" node="k8s-wrk4.foo.bar"
E0325 10:23:15.498523 1 scraper.go:140] "Failed to scrape node" err="request failed, status: \"403 Forbidden\"" node="k8s-wrk2.foo.bar"
E0325 10:23:15.504590 1 scraper.go:140] "Failed to scrape node" err="request failed, status: \"403 Forbidden\"" node="k8s-ctl1.foo.bar"
E0325 10:23:15.504832 1 scraper.go:140] "Failed to scrape node" err="request failed, status: \"403 Forbidden\"" node="k8s-ctl2.foo.bar"
Environment:
- Kubernetes distribution (GKE, EKS, Kubeadm, the hard way, etc.): v1.25.6, vanilla
- Container Network Setup (flannel, calico, etc.): calico
- Kubernetes version (use
kubectl version):
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"", GitTreeState:"", BuildDate:"2023-02-10T12:43:36Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"freebsd/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.6", GitCommit:"ff2c119726cc1f8926fb0585c74b25921e866a28", GitTreeState:"clean", BuildDate:"2023-01-18T19:15:26Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
- Metrics Server manifest
bundled one for 0.7.0 (also tried downgrades til 0.6.1, before realizing this is not the real issue.)
(don't think the rest is useful).
P.S. Shame, but I'm still unable to figure out how to get past 403 error even with the secret/token created manually.
/kind bug
This issue is currently awaiting triage.
If metrics-server contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.
The triage/accepted label can be added by org members by writing /triage accepted in a comment.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
@logicalhan: Closing this issue.
In response to this:
/close 1.24 is no longer a supported kubernetes version.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.