[BUG] metrics server not reporting metrics for hpa

[fragolinux]

What did you do

• How was the cluster created?

  • k3d cluster create test

• What did you do afterwards?

  • installed my pods, i get in events errors about metrics, in particular:
  • FailedComputeMetricsReplicas invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
  • FailedGetResourceMetric failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
  • or failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

What did you expect to happen

hpa getting metrics so it can do its job

Screenshots or terminal output

If applicable, add screenshots or terminal output (code block) to help explain your problem.

Which OS & Architecture

• output of k3d runtime-info

arch: x86_64 cgroupdriver: cgroupfs cgroupversion: "2" endpoint: /var/run/docker.sock filesystem: extfs name: docker os: Docker Desktop ostype: linux version: 20.10.17

Which version of k3d

• output of k3d version

k3d version v5.4.4 k3s version v1.23.8-k3s1 (default)

Which version of docker

• output of docker version and docker info

Client: Cloud integration: v1.0.24 Version: 20.10.17 API version: 1.41 Go version: go1.17.11 Git commit: 100c701 Built: Mon Jun 6 23:04:45 2022 OS/Arch: darwin/amd64 Context: default Experimental: true

Server: Docker Desktop 4.10.1 (82475) Engine: Version: 20.10.17 API version: 1.41 (minimum version 1.12) Go version: go1.17.11 Git commit: a89b842 Built: Mon Jun 6 23:01:23 2022 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.6.6 GitCommit: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1 runc: Version: 1.1.2 GitCommit: v1.1.2-0-ga916309 docker-init: Version: 0.19.0 GitCommit: de40ad0 Client: Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc., v0.8.2) compose: Docker Compose (Docker Inc., v2.6.1) extension: Manages Docker extensions (Docker Inc., v0.2.7) sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0) scan: Docker Scan (Docker Inc., v0.17.0)

Server: Containers: 44 Running: 9 Paused: 0 Stopped: 35 Images: 66 Server Version: 20.10.17 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1 runc version: v1.1.2-0-ga916309 init version: de40ad0 Security Options: seccomp Profile: default cgroupns Kernel Version: 5.10.104-linuxkit Operating System: Docker Desktop OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 2.921GiB Name: docker-desktop ID: IO7B:SU45:NMT4:WQV4:4SFS:NNEJ:SKLF:RA7I:YLWZ:JA7J:EN3U:TSS3 Docker Root Dir: /var/lib/docker Debug Mode: false HTTP Proxy: http.docker.internal:3128 HTTPS Proxy: http.docker.internal:3128 No Proxy: hubproxy.docker.internal Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: e4t.local:5555 hubproxy.docker.internal:5000 127.0.0.0/8 Live Restore Enabled: false

p.s. adding --kubelet-insecure-tls to metrics server deployment didn't help...