metrics-server icon indicating copy to clipboard operation
metrics-server copied to clipboard

Published 20 hours ago • verified publisher icon kubernetes-sigs

metrics-server v0.6.1 : Readiness probe failed: HTTP probe failed with statuscode: 500 with docker desktop and option --kubelet-insecure-tls=true

Open nicoclau opened this issue 2 years ago • 8 comments 

λ kubectl describe pod metrics-server-75d5c7969-ksxrz -n kube-system
Name:                 metrics-server-75d5c7969-ksxrz
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 docker-desktop/192.168.65.4
Start Time:           Mon, 03 Oct 2022 22:28:38 +0200
Labels:               k8s-app=metrics-server
                      pod-template-hash=75d5c7969
Annotations:          kubectl.kubernetes.io/restartedAt: 2022-10-03T18:14:31Z
Status:               Running
IP:                   10.1.0.25
IPs:
  IP:           10.1.0.25
Controlled By:  ReplicaSet/metrics-server-75d5c7969
Containers:
  metrics-server:
    Container ID:  docker://fc9045c8ebd674f5f992e052250462fcf53d3a14867d4c21daec580fa6d7bcc5
    Image:         k8s.gcr.io/metrics-server/metrics-server:v0.6.1
    Image ID:      docker-pullable://k8s.gcr.io/metrics-server/metrics-server@sha256:5ddc6458eb95f5c70bd13fdab90cbd7d6ad1066e5b528ad1dcb28b76c5fb2f00
    Port:          4443/TCP
    Host Port:     4443/TCP
    Args:
      --cert-dir=/tmp
      --secure-port=4443
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --kubelet-use-node-status-port
      --metric-resolution=30s
      --kubelet-insecure-tls=true
    State:          Running
      Started:      Mon, 03 Oct 2022 22:28:39 +0200
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        100m
      memory:     200Mi
    Liveness:     http-get https://:https/livez delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get https://:https/readyz delay=20s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /tmp from tmp-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-v28mg (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  tmp-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-v28mg:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  49s               default-scheduler  Successfully assigned kube-system/metrics-server-75d5c7969-ksxrz to docker-desktop
  Normal   Pulled     48s               kubelet            Container image "k8s.gcr.io/metrics-server/metrics-server:v0.6.1" already present on machine
  Normal   Created    48s               kubelet            Created container metrics-server
  Normal   Started    48s               kubelet            Started container metrics-server
  Warning  Unhealthy  9s (x2 over 19s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500

With the logs

I1003 20:28:40.179101       1 serving.go:342] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I1003 20:28:41.248752       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I1003 20:28:41.248966       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I1003 20:28:41.249009       1 dynamic_serving_content.go:131] "Starting controller" name="serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key"
I1003 20:28:41.249055       1 secure_serving.go:266] Serving securely on [::]:4443
I1003 20:28:41.249116       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
W1003 20:28:41.249240       1 shared_informer.go:372] The sharedIndexInformer has started, run more than once is not allowed
I1003 20:28:41.248777       1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I1003 20:28:41.249397       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1003 20:28:41.248808       1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I1003 20:28:41.249414       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1003 20:28:41.349738       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController 
I1003 20:28:41.349776       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
I1003 20:28:41.349762       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file 
I1003 20:29:08.662960       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I1003 20:29:18.664409       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I1003 20:29:28.661610       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"

Could you tell me what I can do to resolve the problem ? Thanks

/kind support

nicoclau avatar Oct 04 '22 07:10 nicoclau

I tried to see if we can get the metrics of Docker Desktop and it works.

image

λ kubectl get --raw /api/v1/nodes/docker-desktop/proxy/metrics/resource
# HELP container_cpu_usage_seconds_total [ALPHA] Cumulative cpu time consumed by the container in core-seconds
# TYPE container_cpu_usage_seconds_total counter
container_cpu_usage_seconds_total{container="coredns",namespace="kube-system",pod="coredns-6d4b75cb6d-4nn9s"} 116.5846391 1664881001185
container_cpu_usage_seconds_total{container="coredns",namespace="kube-system",pod="coredns-6d4b75cb6d-mwmc7"} 124.9730613 1664881003204
container_cpu_usage_seconds_total{container="dashboard-metrics-scraper",namespace="kubernetes-dashboard",pod="dashboard-metrics-scraper-8c47d4b5d-dkvxx"} 15.7754031 1664881007235
container_cpu_usage_seconds_total{container="etcd",namespace="kube-system",pod="etcd-docker-desktop"} 1504.0063433 1664881013286
container_cpu_usage_seconds_total{container="kube-apiserver",namespace="kube-system",pod="kube-apiserver-docker-desktop"} 3180.4893903 1664881015300
container_cpu_usage_seconds_total{container="kube-controller-manager",namespace="kube-system",pod="kube-controller-manager-docker-desktop"} 1533.6434783 1664881009250
container_cpu_usage_seconds_total{container="kube-proxy",namespace="kube-system",pod="kube-proxy-4g48b"} 21.7684377 1664881005217
container_cpu_usage_seconds_total{container="kube-scheduler",namespace="kube-system",pod="kube-scheduler-docker-desktop"} 296.1239637 1664881011259
container_cpu_usage_seconds_total{container="kubernetes-dashboard",namespace="kubernetes-dashboard",pod="kubernetes-dashboard-6c75475678-5b6m2"} 34.0452856 1664880999176
container_cpu_usage_seconds_total{container="metrics-server",namespace="kube-system",pod="metrics-server-75d5c7969-ksxrz"} 234.7712098 1664880995150
container_cpu_usage_seconds_total{container="storage-provisioner",namespace="kube-system",pod="storage-provisioner"} 139.3906448 1664880997159
container_cpu_usage_seconds_total{container="vpnkit-controller",namespace="kube-system",pod="vpnkit-controller"} 0.1875373 1664880993126
# HELP container_memory_working_set_bytes [ALPHA] Current working set of the container in bytes
# TYPE container_memory_working_set_bytes gauge
container_memory_working_set_bytes{container="coredns",namespace="kube-system",pod="coredns-6d4b75cb6d-4nn9s"} 0 1664881001185
container_memory_working_set_bytes{container="coredns",namespace="kube-system",pod="coredns-6d4b75cb6d-mwmc7"} 0 1664881003204
container_memory_working_set_bytes{container="dashboard-metrics-scraper",namespace="kubernetes-dashboard",pod="dashboard-metrics-scraper-8c47d4b5d-dkvxx"} 0 1664881007235
container_memory_working_set_bytes{container="etcd",namespace="kube-system",pod="etcd-docker-desktop"} 0 1664881013286
container_memory_working_set_bytes{container="kube-apiserver",namespace="kube-system",pod="kube-apiserver-docker-desktop"} 0 1664881015300
container_memory_working_set_bytes{container="kube-controller-manager",namespace="kube-system",pod="kube-controller-manager-docker-desktop"} 0 1664881009250
container_memory_working_set_bytes{container="kube-proxy",namespace="kube-system",pod="kube-proxy-4g48b"} 0 1664881005217
container_memory_working_set_bytes{container="kube-scheduler",namespace="kube-system",pod="kube-scheduler-docker-desktop"} 0 1664881011259
container_memory_working_set_bytes{container="kubernetes-dashboard",namespace="kubernetes-dashboard",pod="kubernetes-dashboard-6c75475678-5b6m2"} 0 1664880999176
container_memory_working_set_bytes{container="metrics-server",namespace="kube-system",pod="metrics-server-75d5c7969-ksxrz"} 0 1664880995150
container_memory_working_set_bytes{container="storage-provisioner",namespace="kube-system",pod="storage-provisioner"} 0 1664880997159
container_memory_working_set_bytes{container="vpnkit-controller",namespace="kube-system",pod="vpnkit-controller"} 0 1664880993126
# HELP container_start_time_seconds [ALPHA] Start time of the container since unix epoch in seconds
# TYPE container_start_time_seconds gauge
container_start_time_seconds{container="coredns",namespace="kube-system",pod="coredns-6d4b75cb6d-4nn9s"} 1.664820675e+09 1664820675000
container_start_time_seconds{container="coredns",namespace="kube-system",pod="coredns-6d4b75cb6d-mwmc7"} 1.664820675e+09 1664820675000
container_start_time_seconds{container="dashboard-metrics-scraper",namespace="kubernetes-dashboard",pod="dashboard-metrics-scraper-8c47d4b5d-dkvxx"} 1.664820675e+09 1664820675000
container_start_time_seconds{container="etcd",namespace="kube-system",pod="etcd-docker-desktop"} 1.664820666e+09 1664820666000
container_start_time_seconds{container="kube-apiserver",namespace="kube-system",pod="kube-apiserver-docker-desktop"} 1.664820666e+09 1664820666000
container_start_time_seconds{container="kube-controller-manager",namespace="kube-system",pod="kube-controller-manager-docker-desktop"} 1.664820666e+09 1664820666000
container_start_time_seconds{container="kube-proxy",namespace="kube-system",pod="kube-proxy-4g48b"} 1.664820675e+09 1664820675000
container_start_time_seconds{container="kube-scheduler",namespace="kube-system",pod="kube-scheduler-docker-desktop"} 1.664820666e+09 1664820666000
container_start_time_seconds{container="kubernetes-dashboard",namespace="kubernetes-dashboard",pod="kubernetes-dashboard-6c75475678-5b6m2"} 1.664820677e+09 1664820677000
container_start_time_seconds{container="metrics-server",namespace="kube-system",pod="metrics-server-75d5c7969-ksxrz"} 1.664828919e+09 1664828919000
container_start_time_seconds{container="storage-provisioner",namespace="kube-system",pod="storage-provisioner"} 1.664820747e+09 1664820747000
container_start_time_seconds{container="vpnkit-controller",namespace="kube-system",pod="vpnkit-controller"} 1.664880752e+09 1664880752000
# HELP node_cpu_usage_seconds_total [ALPHA] Cumulative cpu time consumed by the node in core-seconds
# TYPE node_cpu_usage_seconds_total counter
node_cpu_usage_seconds_total 22625.1735865 1664880978253
# HELP node_memory_working_set_bytes [ALPHA] Current working set of the node in bytes
# TYPE node_memory_working_set_bytes gauge
node_memory_working_set_bytes 2.342232064e+09 1664880978253
# HELP pod_cpu_usage_seconds_total [ALPHA] Cumulative cpu time consumed by the pod in core-seconds
# TYPE pod_cpu_usage_seconds_total counter
pod_cpu_usage_seconds_total{namespace="kube-system",pod="coredns-6d4b75cb6d-4nn9s"} 181.4272158 1664880998770
pod_cpu_usage_seconds_total{namespace="kube-system",pod="coredns-6d4b75cb6d-mwmc7"} 189.6578929 1664881008027
pod_cpu_usage_seconds_total{namespace="kube-system",pod="etcd-docker-desktop"} 2223.1519005 1664881004984
pod_cpu_usage_seconds_total{namespace="kube-system",pod="kube-apiserver-docker-desktop"} 4666.836625 1664881008797
pod_cpu_usage_seconds_total{namespace="kube-system",pod="kube-controller-manager-docker-desktop"} 2241.2116275 1664881006601
pod_cpu_usage_seconds_total{namespace="kube-system",pod="kube-proxy-4g48b"} 34.1695243 1664881011522
pod_cpu_usage_seconds_total{namespace="kube-system",pod="kube-scheduler-docker-desktop"} 427.7361815 1664881002606
pod_cpu_usage_seconds_total{namespace="kube-system",pod="metrics-server-75d5c7969-ksxrz"} 234.8159916 1664880999313
pod_cpu_usage_seconds_total{namespace="kube-system",pod="storage-provisioner"} 198.8464119 1664881009747
pod_cpu_usage_seconds_total{namespace="kube-system",pod="vpnkit-controller"} 32.658104 1664881008572
pod_cpu_usage_seconds_total{namespace="kubernetes-dashboard",pod="dashboard-metrics-scraper-8c47d4b5d-dkvxx"} 21.6286942 1664881012789
pod_cpu_usage_seconds_total{namespace="kubernetes-dashboard",pod="kubernetes-dashboard-6c75475678-5b6m2"} 71.3461952 1664881009614
# HELP pod_memory_working_set_bytes [ALPHA] Current working set of the pod in bytes
# TYPE pod_memory_working_set_bytes gauge
pod_memory_working_set_bytes{namespace="kube-system",pod="coredns-6d4b75cb6d-4nn9s"} 2.49856e+07 1664880998770
pod_memory_working_set_bytes{namespace="kube-system",pod="coredns-6d4b75cb6d-mwmc7"} 2.480128e+07 1664881008027
pod_memory_working_set_bytes{namespace="kube-system",pod="etcd-docker-desktop"} 1.1548672e+08 1664881004984
pod_memory_working_set_bytes{namespace="kube-system",pod="kube-apiserver-docker-desktop"} 4.19844096e+08 1664881008797
pod_memory_working_set_bytes{namespace="kube-system",pod="kube-controller-manager-docker-desktop"} 6.2066688e+07 1664881006601
pod_memory_working_set_bytes{namespace="kube-system",pod="kube-proxy-4g48b"} 2.8086272e+07 1664881011522
pod_memory_working_set_bytes{namespace="kube-system",pod="kube-scheduler-docker-desktop"} 2.7066368e+07 1664881002606
pod_memory_working_set_bytes{namespace="kube-system",pod="metrics-server-75d5c7969-ksxrz"} 2.8274688e+07 1664880999313
pod_memory_working_set_bytes{namespace="kube-system",pod="storage-provisioner"} 1.5290368e+07 1664881009747
pod_memory_working_set_bytes{namespace="kube-system",pod="vpnkit-controller"} 2.7459584e+07 1664881008572
pod_memory_working_set_bytes{namespace="kubernetes-dashboard",pod="dashboard-metrics-scraper-8c47d4b5d-dkvxx"} 2.32448e+07 1664881012789
pod_memory_working_set_bytes{namespace="kubernetes-dashboard",pod="kubernetes-dashboard-6c75475678-5b6m2"} 3.729408e+07 1664881009614
# HELP scrape_error [ALPHA] 1 if there was an error while getting container metrics, 0 otherwise
# TYPE scrape_error gauge
scrape_error 0

nicoclau avatar Oct 04 '22 10:10 nicoclau 


# TYPE container_memory_working_set_bytes gauge
container_memory_working_set_bytes{container="coredns",namespace="kube-system",pod="coredns-6d4b75cb6d-4nn9s"} 0 1664881001185
container_memory_working_set_bytes{container="coredns",namespace="kube-system",pod="coredns-6d4b75cb6d-mwmc7"} 0 1664881003204
container_memory_working_set_bytes{container="dashboard-metrics-scraper",namespace="kubernetes-dashboard",pod="dashboard-metrics-scraper-8c47d4b5d-dkvxx"} 0 1664881007235
container_memory_working_set_bytes{container="etcd",namespace="kube-system",pod="etcd-docker-desktop"} 0 1664881013286
container_memory_working_set_bytes{container="kube-apiserver",namespace="kube-system",pod="kube-apiserver-docker-desktop"} 0 1664881015300
container_memory_working_set_bytes{container="kube-controller-manager",namespace="kube-system",pod="kube-controller-manager-docker-desktop"} 0 1664881009250
container_memory_working_set_bytes{container="kube-proxy",namespace="kube-system",pod="kube-proxy-4g48b"} 0 1664881005217
container_memory_working_set_bytes{container="kube-scheduler",namespace="kube-system",pod="kube-scheduler-docker-desktop"} 0 1664881011259
container_memory_working_set_bytes{container="kubernetes-dashboard",namespace="kubernetes-dashboard",pod="kubernetes-dashboard-6c75475678-5b6m2"} 0 1664880999176
container_memory_working_set_bytes{container="metrics-server",namespace="kube-system",pod="metrics-server-75d5c7969-ksxrz"} 0 1664880995150
container_memory_working_set_bytes{container="storage-provisioner",namespace="kube-system",pod="storage-provisioner"} 0 1664880997159
container_memory_working_set_bytes{container="vpnkit-controller",namespace="kube-system",pod="vpnkit-controller"} 0 1664880993126
#
``

I think this is the problem.
the value of `container_memory_working_set_bytes`  is 0.
You can also confirm with the following command

NODE_NAME=<Name of node in your cluster> kubectl get --raw /api/v1/nodes/$NODE_NAME/proxy/stats/summary


metrics-server did not get memory data from kubelet's `stats/summary` endpoint.

yangjunmyfm192085 avatar Oct 04 '22 12:10 yangjunmyfm192085

Thanks for giving me the direction @yangjunmyfm192085 ! What can I do to solve this problem?

nicoclau avatar Oct 04 '22 13:10 nicoclau

It seems the stats/summary endpoint of kubelet does not get memory statistics Is it similar to the issue https://github.com/kubernetes-sigs/metrics-server/issues/1036 ?

yangjunmyfm192085 avatar Oct 04 '22 14:10 yangjunmyfm192085

I get this with λ kubectl get --raw /api/v1/nodes/docker-desktop/proxy/stats/summary :

{
 "node": {
  "nodeName": "docker-desktop",
  "systemContainers": [
   {
    "name": "pods",
    "startTime": "2022-10-04T18:59:37Z",
    "cpu": {
     "time": "2022-10-04T19:05:31Z",
     "usageNanoCores": 95100598,
     "usageCoreNanoSeconds": 59055745700
    },
    "memory": {
     "time": "2022-10-04T19:05:31Z",
     "availableBytes": 12714287104,
     "usageBytes": 1138167808,
     "workingSetBytes": 629800960,
     "rssBytes": 513200128,
     "pageFaults": 0,
     "majorPageFaults": 0
    }
   },
   {
    "name": "kubelet",
    "startTime": "2022-10-04T18:59:37Z",
    "cpu": {
     "time": "2022-10-04T19:05:28Z",
     "usageNanoCores": 95826042,
     "usageCoreNanoSeconds": 55652055500
    },
    "memory": {
     "time": "2022-10-04T19:05:28Z",
     "usageBytes": 847597568,
     "workingSetBytes": 539414528,
     "rssBytes": 313708544,
     "pageFaults": 1020030,
     "majorPageFaults": 2409
    }
   }
  ],
  "startTime": "2022-10-04T18:59:11Z",
  "cpu": {
   "time": "2022-10-04T19:05:00Z",
   "usageNanoCores": 308326092,
   "usageCoreNanoSeconds": 125011715900
  },
  "memory": {
   "time": "2022-10-04T19:05:00Z",
   "availableBytes": 11577896960,
   "usageBytes": 3246280704,
   "workingSetBytes": 1766191104,
   "rssBytes": 1066156032,
   "pageFaults": 89925,
   "majorPageFaults": 1782
  },
  "network": {
   "time": "2022-10-04T19:05:00Z",
   "name": "eth0",
   "rxBytes": 26089,
   "rxErrors": 0,
   "txBytes": 13193,
   "txErrors": 0,
   "interfaces": [
    {
     "name": "tunl0",
     "rxBytes": 0,
     "rxErrors": 0,
     "txBytes": 0,
     "txErrors": 0
    },
    {
     "name": "sit0",
     "rxBytes": 0,
     "rxErrors": 0,
     "txBytes": 0,
     "txErrors": 0
    },
    {
     "name": "services1",
     "rxBytes": 16227,
     "rxErrors": 0,
     "txBytes": 13864,
     "txErrors": 0
    },
    {
     "name": "eth0",
     "rxBytes": 26089,
     "rxErrors": 0,
     "txBytes": 13193,
     "txErrors": 0
    },
    {
     "name": "cni0",
     "rxBytes": 671478,
     "rxErrors": 0,
     "txBytes": 901524,
     "txErrors": 0
    }
   ]
  },
  "fs": {
   "time": "2022-10-04T19:05:00Z",
   "availableBytes": 251505799168,
   "capacityBytes": 269490393088,
   "usedBytes": 4223922176,
   "inodesFree": 16737806,
   "inodes": 16777216,
   "inodesUsed": 39410
  },
  "runtime": {
   "imageFs": {
    "time": "1970-01-01T00:00:01Z",
    "availableBytes": 251505799168,
    "capacityBytes": 269490393088,
    "usedBytes": 1434797,
    "inodesFree": 16737806,
    "inodes": 16777216,
    "inodesUsed": 534
   }
  },
  "rlimit": {
   "time": "2022-10-04T19:05:32Z",
   "maxpid": 32768,
   "curproc": 1018
  }
 },
 "pods": [
  {
   "podRef": {
    "name": "storage-provisioner",
    "namespace": "kube-system",
    "uid": "10fcec4f-2347-44bb-9729-c18ae09b10f8"
   },
   "startTime": "2022-10-04T18:59:46Z",
   "containers": [
    {
     "name": "storage-provisioner",
     "startTime": "2022-10-04T19:00:02Z",
     "cpu": {
      "time": "2022-10-04T19:05:12Z",
      "usageNanoCores": 2603352,
      "usageCoreNanoSeconds": 828543100
     },
     "memory": {
      "time": "2022-10-04T19:05:12Z",
      "workingSetBytes": 0
     },
     "rootfs": {
      "time": "2022-10-04T19:05:12Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216
     },
     "logs": {
      "time": "2022-10-04T19:05:32Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216,
      "inodesUsed": 0
     }
    }
   ],
   "cpu": {
    "time": "2022-10-04T19:05:24Z",
    "usageNanoCores": 3175446,
    "usageCoreNanoSeconds": 918079500
   },
   "memory": {
    "time": "2022-10-04T19:05:24Z",
    "usageBytes": 32727040,
    "workingSetBytes": 7856128,
    "rssBytes": 8654848,
    "pageFaults": 0,
    "majorPageFaults": 0
   },
   "volume": [
    {
     "time": "2022-10-04T19:04:10Z",
     "availableBytes": 13239218176,
     "capacityBytes": 13239230464,
     "usedBytes": 12288,
     "inodesFree": 1628908,
     "inodes": 1628917,
     "inodesUsed": 9,
     "name": "kube-api-access-lk68z"
    }
   ],
   "ephemeral-storage": {
    "time": "2022-10-04T19:05:32Z",
    "availableBytes": 251505799168,
    "capacityBytes": 269490393088,
    "usedBytes": 4096,
    "inodesFree": 16737806,
    "inodes": 16777216,
    "inodesUsed": 1
   },
   "process_stats": {
    "process_count": 0
   }
  },
  {
   "podRef": {
    "name": "dashboard-metrics-scraper-8c47d4b5d-dkvxx",
    "namespace": "kubernetes-dashboard",
    "uid": "e6c517d8-8f02-4661-aec9-d18c8e9b5100"
   },
   "startTime": "2022-10-04T18:59:46Z",
   "containers": [
    {
     "name": "dashboard-metrics-scraper",
     "startTime": "2022-10-04T18:59:51Z",
     "cpu": {
      "time": "2022-10-04T19:05:14Z",
      "usageNanoCores": 94504,
      "usageCoreNanoSeconds": 132573000
     },
     "memory": {
      "time": "2022-10-04T19:05:14Z",
      "workingSetBytes": 0
     },
     "rootfs": {
      "time": "2022-10-04T19:05:14Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216
     },
     "logs": {
      "time": "2022-10-04T19:05:32Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216,
      "inodesUsed": 0
     }
    }
   ],
   "cpu": {
    "time": "2022-10-04T19:05:17Z",
    "usageNanoCores": 80755,
    "usageCoreNanoSeconds": 164440000
   },
   "memory": {
    "time": "2022-10-04T19:05:17Z",
    "usageBytes": 31289344,
    "workingSetBytes": 7634944,
    "rssBytes": 6205440,
    "pageFaults": 0,
    "majorPageFaults": 0
   },
   "volume": [
    {
     "time": "2022-10-04T19:05:27Z",
     "availableBytes": 251505795072,
     "capacityBytes": 269490393088,
     "usedBytes": 16384,
     "inodesFree": 16737806,
     "inodes": 16777216,
     "inodesUsed": 2,
     "name": "tmp-volume"
    },
    {
     "time": "2022-10-04T19:05:27Z",
     "availableBytes": 13239218176,
     "capacityBytes": 13239230464,
     "usedBytes": 12288,
     "inodesFree": 1628908,
     "inodes": 1628917,
     "inodesUsed": 9,
     "name": "kube-api-access-ghtnm"
    }
   ],
   "ephemeral-storage": {
    "time": "2022-10-04T19:05:32Z",
    "availableBytes": 251505799168,
    "capacityBytes": 269490393088,
    "usedBytes": 20480,
    "inodesFree": 16737806,
    "inodes": 16777216,
    "inodesUsed": 3
   },
   "process_stats": {
    "process_count": 0
   }
  },
  {
   "podRef": {
    "name": "kubernetes-dashboard-6c75475678-5b6m2",
    "namespace": "kubernetes-dashboard",
    "uid": "512f38dc-b821-4e24-9b37-eddaaf53d781"
   },
   "startTime": "2022-10-04T18:59:47Z",
   "containers": [
    {
     "name": "kubernetes-dashboard",
     "startTime": "2022-10-04T18:59:49Z",
     "cpu": {
      "time": "2022-10-04T19:05:16Z",
      "usageNanoCores": 319837,
      "usageCoreNanoSeconds": 459275500
     },
     "memory": {
      "time": "2022-10-04T19:05:16Z",
      "workingSetBytes": 0
     },
     "rootfs": {
      "time": "2022-10-04T19:05:16Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216
     },
     "logs": {
      "time": "2022-10-04T19:05:32Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216,
      "inodesUsed": 0
     }
    }
   ],
   "cpu": {
    "time": "2022-10-04T19:05:28Z",
    "usageNanoCores": 459359,
    "usageCoreNanoSeconds": 507896100
   },
   "memory": {
    "time": "2022-10-04T19:05:28Z",
    "usageBytes": 41553920,
    "workingSetBytes": 10735616,
    "rssBytes": 9326592,
    "pageFaults": 0,
    "majorPageFaults": 0
   },
   "volume": [
    {
     "time": "2022-10-04T19:04:07Z",
     "availableBytes": 251505799168,
     "capacityBytes": 269490393088,
     "usedBytes": 1216512,
     "inodesFree": 16737806,
     "inodes": 16777216,
     "inodesUsed": 9,
     "name": "tmp-volume"
    },
    {
     "time": "2022-10-04T19:04:07Z",
     "availableBytes": 13239230464,
     "capacityBytes": 13239230464,
     "usedBytes": 0,
     "inodesFree": 1628914,
     "inodes": 1628917,
     "inodesUsed": 3,
     "name": "kubernetes-dashboard-certs"
    },
    {
     "time": "2022-10-04T19:04:07Z",
     "availableBytes": 13239218176,
     "capacityBytes": 13239230464,
     "usedBytes": 12288,
     "inodesFree": 1628908,
     "inodes": 1628917,
     "inodesUsed": 9,
     "name": "kube-api-access-rdhwd"
    }
   ],
   "ephemeral-storage": {
    "time": "2022-10-04T19:05:32Z",
    "availableBytes": 251505799168,
    "capacityBytes": 269490393088,
    "usedBytes": 1220608,
    "inodesFree": 16737806,
    "inodes": 16777216,
    "inodesUsed": 10
   },
   "process_stats": {
    "process_count": 0
   }
  },
  {
   "podRef": {
    "name": "kube-proxy-4g48b",
    "namespace": "kube-system",
    "uid": "6117e0ae-1471-40c3-9004-12977aa0260c"
   },
   "startTime": "2022-10-04T18:59:47Z",
   "containers": [
    {
     "name": "kube-proxy",
     "startTime": "2022-10-04T18:59:48Z",
     "cpu": {
      "time": "2022-10-04T19:05:20Z",
      "usageNanoCores": 449155,
      "usageCoreNanoSeconds": 800989900
     },
     "memory": {
      "time": "2022-10-04T19:05:20Z",
      "workingSetBytes": 0
     },
     "rootfs": {
      "time": "2022-10-04T19:05:20Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 2294,
      "inodesFree": 16737806,
      "inodes": 16777216
     },
     "logs": {
      "time": "2022-10-04T19:05:32Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216,
      "inodesUsed": 0
     }
    }
   ],
   "cpu": {
    "time": "2022-10-04T19:05:19Z",
    "usageNanoCores": 67774,
    "usageCoreNanoSeconds": 819859800
   },
   "memory": {
    "time": "2022-10-04T19:05:19Z",
    "usageBytes": 57909248,
    "workingSetBytes": 20602880,
    "rssBytes": 11259904,
    "pageFaults": 0,
    "majorPageFaults": 0
   },
   "volume": [
    {
     "time": "2022-10-04T19:05:24Z",
     "availableBytes": 251505795072,
     "capacityBytes": 269490393088,
     "usedBytes": 16384,
     "inodesFree": 16737806,
     "inodes": 16777216,
     "inodesUsed": 7,
     "name": "kube-proxy"
    },
    {
     "time": "2022-10-04T19:05:24Z",
     "availableBytes": 13239218176,
     "capacityBytes": 13239230464,
     "usedBytes": 12288,
     "inodesFree": 1628908,
     "inodes": 1628917,
     "inodesUsed": 9,
     "name": "kube-api-access-ttg2t"
    }
   ],
   "ephemeral-storage": {
    "time": "2022-10-04T19:05:32Z",
    "availableBytes": 251505799168,
    "capacityBytes": 269490393088,
    "usedBytes": 22774,
    "inodesFree": 16737806,
    "inodes": 16777216,
    "inodesUsed": 8
   },
   "process_stats": {
    "process_count": 0
   }
  },
  {
   "podRef": {
    "name": "coredns-6d4b75cb6d-4nn9s",
    "namespace": "kube-system",
    "uid": "592d0e70-a57b-4785-9283-5aad12f1d658"
   },
   "startTime": "2022-10-04T18:59:47Z",
   "containers": [
    {
     "name": "coredns",
     "startTime": "2022-10-04T18:59:48Z",
     "cpu": {
      "time": "2022-10-04T19:05:24Z",
      "usageNanoCores": 2233658,
      "usageCoreNanoSeconds": 868172900
     },
     "memory": {
      "time": "2022-10-04T19:05:24Z",
      "workingSetBytes": 0
     },
     "rootfs": {
      "time": "2022-10-04T19:05:24Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216
     },
     "logs": {
      "time": "2022-10-04T19:05:32Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216,
      "inodesUsed": 0
     }
    }
   ],
   "cpu": {
    "time": "2022-10-04T19:05:32Z",
    "usageNanoCores": 1645921,
    "usageCoreNanoSeconds": 918197600
   },
   "memory": {
    "time": "2022-10-04T19:05:32Z",
    "availableBytes": 162721792,
    "usageBytes": 50274304,
    "workingSetBytes": 15536128,
    "rssBytes": 12910592,
    "pageFaults": 0,
    "majorPageFaults": 0
   },
   "volume": [
    {
     "time": "2022-10-04T19:04:25Z",
     "availableBytes": 251505799168,
     "capacityBytes": 269490393088,
     "usedBytes": 12288,
     "inodesFree": 16737806,
     "inodes": 16777216,
     "inodesUsed": 5,
     "name": "config-volume"
    },
    {
     "time": "2022-10-04T19:04:25Z",
     "availableBytes": 178245632,
     "capacityBytes": 178257920,
     "usedBytes": 12288,
     "inodesFree": 1628908,
     "inodes": 1628917,
     "inodesUsed": 9,
     "name": "kube-api-access-cgfxn"
    }
   ],
   "ephemeral-storage": {
    "time": "2022-10-04T19:05:32Z",
    "availableBytes": 251505799168,
    "capacityBytes": 269490393088,
    "usedBytes": 16384,
    "inodesFree": 16737806,
    "inodes": 16777216,
    "inodesUsed": 6
   },
   "process_stats": {
    "process_count": 0
   }
  },
  {
   "podRef": {
    "name": "kube-scheduler-docker-desktop",
    "namespace": "kube-system",
    "uid": "1fa929fab6e17047f7779b2ab8125174"
   },
   "startTime": "2022-10-04T18:59:39Z",
   "containers": [
    {
     "name": "kube-scheduler",
     "startTime": "2022-10-04T18:59:39Z",
     "cpu": {
      "time": "2022-10-04T19:05:26Z",
      "usageNanoCores": 5292680,
      "usageCoreNanoSeconds": 3579979300
     },
     "memory": {
      "time": "2022-10-04T19:05:26Z",
      "workingSetBytes": 0
     },
     "rootfs": {
      "time": "2022-10-04T19:05:26Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216
     },
     "logs": {
      "time": "2022-10-04T19:05:32Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216,
      "inodesUsed": 0
     }
    }
   ],
   "cpu": {
    "time": "2022-10-04T19:05:19Z",
    "usageNanoCores": 4090310,
    "usageCoreNanoSeconds": 3586612500
   },
   "memory": {
    "time": "2022-10-04T19:05:19Z",
    "usageBytes": 59740160,
    "workingSetBytes": 18243584,
    "rssBytes": 16220160,
    "pageFaults": 0,
    "majorPageFaults": 0
   },
   "ephemeral-storage": {
    "time": "2022-10-04T19:05:32Z",
    "availableBytes": 251505799168,
    "capacityBytes": 269490393088,
    "usedBytes": 4096,
    "inodesFree": 16737806,
    "inodes": 16777216,
    "inodesUsed": 1
   },
   "process_stats": {
    "process_count": 0
   }
  },
  {
   "podRef": {
    "name": "kube-apiserver-docker-desktop",
    "namespace": "kube-system",
    "uid": "f76ea91a200a6b1cfe31c7a114460aac"
   },
   "startTime": "2022-10-04T18:59:38Z",
   "containers": [
    {
     "name": "kube-apiserver",
     "startTime": "2022-10-04T18:59:39Z",
     "cpu": {
      "time": "2022-10-04T19:05:30Z",
      "usageNanoCores": 64835292,
      "usageCoreNanoSeconds": 24717260200
     },
     "memory": {
      "time": "2022-10-04T19:05:30Z",
      "workingSetBytes": 0
     },
     "rootfs": {
      "time": "2022-10-04T19:05:30Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216
     },
     "logs": {
      "time": "2022-10-04T19:05:32Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216,
      "inodesUsed": 0
     }
    }
   ],
   "cpu": {
    "time": "2022-10-04T19:05:23Z",
    "usageNanoCores": 43288307,
    "usageCoreNanoSeconds": 24479856500
   },
   "memory": {
    "time": "2022-10-04T19:05:23Z",
    "usageBytes": 424861696,
    "workingSetBytes": 338219008,
    "rssBytes": 320299008,
    "pageFaults": 0,
    "majorPageFaults": 0
   },
   "ephemeral-storage": {
    "time": "2022-10-04T19:05:32Z",
    "availableBytes": 251505799168,
    "capacityBytes": 269490393088,
    "usedBytes": 4096,
    "inodesFree": 16737806,
    "inodes": 16777216,
    "inodesUsed": 1
   },
   "process_stats": {
    "process_count": 0
   }
  },
  {
   "podRef": {
    "name": "vpnkit-controller",
    "namespace": "kube-system",
    "uid": "4aa8d179-998a-4a68-a957-6696347d10c8"
   },
   "startTime": "2022-10-04T18:59:47Z",
   "containers": [
    {
     "name": "vpnkit-controller",
     "startTime": "2022-10-04T19:00:21Z",
     "cpu": {
      "time": "2022-10-04T19:05:10Z",
      "usageNanoCores": 330687,
      "usageCoreNanoSeconds": 156389900
     },
     "memory": {
      "time": "2022-10-04T19:05:10Z",
      "workingSetBytes": 0
     },
     "rootfs": {
      "time": "2022-10-04T19:05:10Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216
     },
     "logs": {
      "time": "2022-10-04T19:05:32Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216,
      "inodesUsed": 0
     }
    }
   ],
   "cpu": {
    "time": "2022-10-04T19:05:16Z",
    "usageNanoCores": 0,
    "usageCoreNanoSeconds": 300983800
   },
   "memory": {
    "time": "2022-10-04T19:05:16Z",
    "usageBytes": 26542080,
    "workingSetBytes": 8970240,
    "rssBytes": 7344128,
    "pageFaults": 0,
    "majorPageFaults": 0
   },
   "volume": [
    {
     "time": "2022-10-04T19:04:01Z",
     "availableBytes": 13239218176,
     "capacityBytes": 13239230464,
     "usedBytes": 12288,
     "inodesFree": 1628908,
     "inodes": 1628917,
     "inodesUsed": 9,
     "name": "kube-api-access-svt8j"
    }
   ],
   "ephemeral-storage": {
    "time": "2022-10-04T19:05:32Z",
    "availableBytes": 251505799168,
    "capacityBytes": 269490393088,
    "usedBytes": 4096,
    "inodesFree": 16737806,
    "inodes": 16777216,
    "inodesUsed": 1
   },
   "process_stats": {
    "process_count": 0
   }
  },
  {
   "podRef": {
    "name": "coredns-6d4b75cb6d-mwmc7",
    "namespace": "kube-system",
    "uid": "f4eb3d2d-ab2c-417e-9c9f-468613bc848c"
   },
   "startTime": "2022-10-04T18:59:47Z",
   "containers": [
    {
     "name": "coredns",
     "startTime": "2022-10-04T18:59:48Z",
     "cpu": {
      "time": "2022-10-04T19:05:22Z",
      "usageNanoCores": 2033844,
      "usageCoreNanoSeconds": 815545300
     },
     "memory": {
      "time": "2022-10-04T19:05:22Z",
      "workingSetBytes": 0
     },
     "rootfs": {
      "time": "2022-10-04T19:05:22Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216
     },
     "logs": {
      "time": "2022-10-04T19:05:32Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216,
      "inodesUsed": 0
     }
    }
   ],
   "cpu": {
    "time": "2022-10-04T19:05:14Z",
    "usageNanoCores": 1485137,
    "usageCoreNanoSeconds": 833213200
   },
   "memory": {
    "time": "2022-10-04T19:05:14Z",
    "availableBytes": 163815424,
    "usageBytes": 18497536,
    "workingSetBytes": 14442496,
    "rssBytes": 12926976,
    "pageFaults": 0,
    "majorPageFaults": 0
   },
   "volume": [
    {
     "time": "2022-10-04T19:05:25Z",
     "availableBytes": 251505795072,
     "capacityBytes": 269490393088,
     "usedBytes": 12288,
     "inodesFree": 16737806,
     "inodes": 16777216,
     "inodesUsed": 5,
     "name": "config-volume"
    },
    {
     "time": "2022-10-04T19:05:25Z",
     "availableBytes": 178245632,
     "capacityBytes": 178257920,
     "usedBytes": 12288,
     "inodesFree": 1628908,
     "inodes": 1628917,
     "inodesUsed": 9,
     "name": "kube-api-access-t9lz7"
    }
   ],
   "ephemeral-storage": {
    "time": "2022-10-04T19:05:32Z",
    "availableBytes": 251505799168,
    "capacityBytes": 269490393088,
    "usedBytes": 16384,
    "inodesFree": 16737806,
    "inodes": 16777216,
    "inodesUsed": 6
   },
   "process_stats": {
    "process_count": 0
   }
  },
  {
   "podRef": {
    "name": "kube-controller-manager-docker-desktop",
    "namespace": "kube-system",
    "uid": "d9c6bbd179b0d64e8d303a659acf3a74"
   },
   "startTime": "2022-10-04T18:59:39Z",
   "containers": [
    {
     "name": "kube-controller-manager",
     "startTime": "2022-10-04T18:59:39Z",
     "cpu": {
      "time": "2022-10-04T19:05:28Z",
      "usageNanoCores": 32801136,
      "usageCoreNanoSeconds": 11991281700
     },
     "memory": {
      "time": "2022-10-04T19:05:28Z",
      "workingSetBytes": 0
     },
     "rootfs": {
      "time": "2022-10-04T19:05:28Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216
     },
     "logs": {
      "time": "2022-10-04T19:05:32Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216,
      "inodesUsed": 0
     }
    }
   ],
   "cpu": {
    "time": "2022-10-04T19:05:27Z",
    "usageNanoCores": 20546980,
    "usageCoreNanoSeconds": 11996983600
   },
   "memory": {
    "time": "2022-10-04T19:05:27Z",
    "usageBytes": 130772992,
    "workingSetBytes": 55619584,
    "rssBytes": 45662208,
    "pageFaults": 0,
    "majorPageFaults": 0
   },
   "ephemeral-storage": {
    "time": "2022-10-04T19:05:32Z",
    "availableBytes": 251505799168,
    "capacityBytes": 269490393088,
    "usedBytes": 4096,
    "inodesFree": 16737806,
    "inodes": 16777216,
    "inodesUsed": 1
   },
   "process_stats": {
    "process_count": 0
   }
  },
  {
   "podRef": {
    "name": "etcd-docker-desktop",
    "namespace": "kube-system",
    "uid": "2449ddc0985e3be8dd23ffc4d12cb53b"
   },
   "startTime": "2022-10-04T18:59:38Z",
   "containers": [
    {
     "name": "etcd",
     "startTime": "2022-10-04T18:59:39Z",
     "cpu": {
      "time": "2022-10-04T19:05:32Z",
      "usageNanoCores": 25412642,
      "usageCoreNanoSeconds": 11298633800
     },
     "memory": {
      "time": "2022-10-04T19:05:32Z",
      "workingSetBytes": 0
     },
     "rootfs": {
      "time": "2022-10-04T19:05:32Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216
     },
     "logs": {
      "time": "2022-10-04T19:05:32Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216,
      "inodesUsed": 0
     }
    }
   ],
   "cpu": {
    "time": "2022-10-04T19:05:31Z",
    "usageNanoCores": 17298376,
    "usageCoreNanoSeconds": 11310856900
   },
   "memory": {
    "time": "2022-10-04T19:05:31Z",
    "usageBytes": 203706368,
    "workingSetBytes": 112197632,
    "rssBytes": 47185920,
    "pageFaults": 0,
    "majorPageFaults": 0
   },
   "ephemeral-storage": {
    "time": "2022-10-04T19:05:32Z",
    "availableBytes": 251505799168,
    "capacityBytes": 269490393088,
    "usedBytes": 4096,
    "inodesFree": 16737806,
    "inodes": 16777216,
    "inodesUsed": 1
   },
   "process_stats": {
    "process_count": 0
   }
  },
  {
   "podRef": {
    "name": "metrics-server-75d5c7969-ksxrz",
    "namespace": "kube-system",
    "uid": "dfec7ba3-0770-47f5-adbd-bebc4b0478f0"
   },
   "startTime": "2022-10-04T18:59:47Z",
   "containers": [
    {
     "name": "metrics-server",
     "startTime": "2022-10-04T18:59:48Z",
     "cpu": {
      "time": "2022-10-04T19:05:18Z",
      "usageNanoCores": 4939394,
      "usageCoreNanoSeconds": 2678142400
     },
     "memory": {
      "time": "2022-10-04T19:05:18Z",
      "workingSetBytes": 0
     },
     "rootfs": {
      "time": "2022-10-04T19:05:18Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216
     },
     "logs": {
      "time": "2022-10-04T19:05:32Z",
      "availableBytes": 251505799168,
      "capacityBytes": 269490393088,
      "usedBytes": 0,
      "inodesFree": 16737806,
      "inodes": 16777216,
      "inodesUsed": 0
     }
    }
   ],
   "cpu": {
    "time": "2022-10-04T19:05:27Z",
    "usageNanoCores": 3619560,
    "usageCoreNanoSeconds": 2740932200
   },
   "memory": {
    "time": "2022-10-04T19:05:27Z",
    "usageBytes": 60133376,
    "workingSetBytes": 19582976,
    "rssBytes": 15204352,
    "pageFaults": 0,
    "majorPageFaults": 0
   },
   "volume": [
    {
     "time": "2022-10-04T19:05:05Z",
     "availableBytes": 251505799168,
     "capacityBytes": 269490393088,
     "usedBytes": 12288,
     "inodesFree": 16737806,
     "inodes": 16777216,
     "inodesUsed": 3,
     "name": "tmp-dir"
    },
    {
     "time": "2022-10-04T19:05:05Z",
     "availableBytes": 13239218176,
     "capacityBytes": 13239230464,
     "usedBytes": 12288,
     "inodesFree": 1628908,
     "inodes": 1628917,
     "inodesUsed": 9,
     "name": "kube-api-access-v28mg"
    }
   ],
   "ephemeral-storage": {
    "time": "2022-10-04T19:05:32Z",
    "availableBytes": 251505799168,
    "capacityBytes": 269490393088,
    "usedBytes": 16384,
    "inodesFree": 16737806,
    "inodes": 16777216,
    "inodesUsed": 4
   },
   "process_stats": {
    "process_count": 0
   }
  }
 ]
}

nicoclau avatar Oct 04 '22 19:10 nicoclau 

 "memory": {
      "time": "2022-10-04T19:05:18Z",
      "workingSetBytes": 0
     },

the same issue

yangjunmyfm192085 avatar Oct 04 '22 23:10 yangjunmyfm192085

I tried with Minikube and still to no avail :(

 kubectl get --raw /api/v1/nodes/minikube/proxy/stats/summary
{
 "node": {
  "nodeName": "minikube",
  "systemContainers": [
   {
    "name": "kubelet",
    "startTime": "2022-10-06T15:56:01Z",
    "cpu": {
     "time": "2022-10-06T15:56:01Z",
     "usageNanoCores": 0,
     "usageCoreNanoSeconds": 0
    },
    "memory": {
     "time": "2022-10-06T15:56:01Z",
     "usageBytes": 114909184,
     "workingSetBytes": 33673216,
     "rssBytes": 29732864,
     "pageFaults": 7755,
     "majorPageFaults": 528
    }
   },
   {
    "name": "runtime",
    "startTime": "2022-10-06T15:55:49Z",
    "cpu": {
     "time": "2022-10-06T15:56:01Z",
     "usageNanoCores": 0,
     "usageCoreNanoSeconds": 0
    },
    "memory": {
     "time": "2022-10-06T15:56:01Z",
     "usageBytes": 55218176,
     "workingSetBytes": 54542336,
     "rssBytes": 52011008,
     "pageFaults": 32967,
     "majorPageFaults": 0
    }
   },
   {
    "name": "pods",
    "startTime": "2022-10-06T15:56:01Z",
    "cpu": {
     "time": "2022-10-06T16:03:23Z",
     "usageNanoCores": 97916797,
     "usageCoreNanoSeconds": 93816632700
    },
    "memory": {
     "time": "2022-10-06T16:03:23Z",
     "availableBytes": 12773404672,
     "usageBytes": 1022820352,
     "workingSetBytes": 570683392,
     "rssBytes": 471523328,
     "pageFaults": 0,
     "majorPageFaults": 0
    }
   }
  ],
  "startTime": "2022-10-06T15:55:09Z",
  "cpu": {
   "time": "2022-10-06T16:03:23Z",
   "usageNanoCores": 140348378,
   "usageCoreNanoSeconds": 136635717800
  },
  "memory": {
   "time": "2022-10-06T16:03:23Z",
   "availableBytes": 12297957376,
   "usageBytes": 1769955328,
   "workingSetBytes": 1046130688,
   "rssBytes": 702144512,
   "pageFaults": 40491,
   "majorPageFaults": 165
  },
  "network": {
   "time": "2022-10-06T16:03:23Z",
   "name": "eth0",
   "rxBytes": 389705,
   "rxErrors": 0,
   "txBytes": 1904451,
   "txErrors": 0,
   "interfaces": [
    {
     "name": "tunl0",
     "rxBytes": 0,
     "rxErrors": 0,
     "txBytes": 0,
     "txErrors": 0
    },
    {
     "name": "sit0",
     "rxBytes": 0,
     "rxErrors": 0,
     "txBytes": 0,
     "txErrors": 0
    },
    {
     "name": "eth0",
     "rxBytes": 389705,
     "rxErrors": 0,
     "txBytes": 1904451,
     "txErrors": 0
    }
   ]
  },
  "fs": {
   "time": "2022-10-06T16:03:23Z",
   "availableBytes": 239631134720,
   "capacityBytes": 269490393088,
   "usedBytes": 16098586624,
   "inodesFree": 16504036,
   "inodes": 16777216,
   "inodesUsed": 273180
  },
  "runtime": {
   "imageFs": {
    "time": "2022-10-06T16:03:23Z",
    "availableBytes": 239631134720,
    "capacityBytes": 269490393088,
    "usedBytes": 1156495716,
    "inodesFree": 16504036,
    "inodes": 16777216,
    "inodesUsed": 273180
   }
  },
  "rlimit": {
   "time": "2022-10-06T16:03:29Z",
   "maxpid": 4194304,
   "curproc": 967
  }
 },
 "pods": []
}

I had the events:

kubectl describe po metrics-server-7c57dd9c77-7cmwc -n kube-system
Name:                 metrics-server-7c57dd9c77-7cmwc
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 minikube/192.168.49.2
Start Time:           Wed, 05 Oct 2022 17:42:22 +0200
Labels:               k8s-app=metrics-server
                      pod-template-hash=7c57dd9c77
Annotations:          <none>
Status:               Running
IP:                   172.17.0.3
IPs:
  IP:           172.17.0.3
Controlled By:  ReplicaSet/metrics-server-7c57dd9c77
Containers:
  metrics-server:
    Container ID:  docker://b42c61d93d1bbbd603c325d8c7163c9e3ae070206df0e1f938f3bc61169a0674
    Image:         k8s.gcr.io/metrics-server/metrics-server:v0.6.1@sha256:5ddc6458eb95f5c70bd13fdab90cbd7d6ad1066e5b528ad1dcb28b76c5fb2f00
    Image ID:      docker-pullable://k8s.gcr.io/metrics-server/metrics-server@sha256:5ddc6458eb95f5c70bd13fdab90cbd7d6ad1066e5b528ad1dcb28b76c5fb2f00
    Port:          4443/TCP
    Host Port:     0/TCP
    Args:
      --cert-dir=/tmp
      --secure-port=4443
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --kubelet-use-node-status-port
      --metric-resolution=60s
      --kubelet-insecure-tls
    State:          Running
      Started:      Thu, 06 Oct 2022 17:56:36 +0200
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Thu, 06 Oct 2022 17:56:12 +0200
      Finished:     Thu, 06 Oct 2022 17:56:36 +0200
    Ready:          True
    Restart Count:  3
    Requests:
      cpu:        100m
      memory:     200Mi
    Liveness:     http-get https://:https/livez delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get https://:https/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /tmp from tmp-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-k54wr (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  tmp-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-k54wr:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason          Age                   From               Message
  ----     ------          ----                  ----               -------
  Normal   Scheduled       24h                   default-scheduler  Successfully assigned kube-system/metrics-server-7c57dd9c77-7cmwc to minikube
  Normal   Pulling         24h                   kubelet            Pulling image "k8s.gcr.io/metrics-server/metrics-server:v0.6.1@sha256:5ddc6458eb95f5c70bd13fdab90cbd7d6ad1066e5b528ad1dcb28b76c5fb2f00"
  Normal   Pulled          24h                   kubelet            Successfully pulled image "k8s.gcr.io/metrics-server/metrics-server:v0.6.1@sha256:5ddc6458eb95f5c70bd13fdab90cbd7d6ad1066e5b528ad1dcb28b76c5fb2f00" in 16.3926953s
  Normal   Created         24h                   kubelet            Created container metrics-server
  Normal   Started         24h                   kubelet            Started container metrics-server
  Warning  Unhealthy       24h                   kubelet            Readiness probe failed: Get "https://172.17.0.5:4443/readyz": dial tcp 172.17.0.5:4443: connect: connection refused
  Warning  Unhealthy       24h (x8 over 24h)     kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500
  Warning  Unhealthy       24h                   kubelet            Liveness probe failed: Get "https://172.17.0.5:4443/livez": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy       24h                   kubelet            Readiness probe failed: Get "https://172.17.0.5:4443/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Normal   SandboxChanged  24h                   kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled          24h                   kubelet            Container image "k8s.gcr.io/metrics-server/metrics-server:v0.6.1@sha256:5ddc6458eb95f5c70bd13fdab90cbd7d6ad1066e5b528ad1dcb28b76c5fb2f00" already present on machine
  Normal   Created         24h                   kubelet            Created container metrics-server
  Normal   Started         24h                   kubelet            Started container metrics-server
  Warning  Unhealthy       24h                   kubelet            Liveness probe failed: Get "https://172.17.0.5:4443/livez": dial tcp 172.17.0.5:4443: connect: connection refused
  Warning  Unhealthy       24h (x2 over 24h)     kubelet            Readiness probe failed: Get "https://172.17.0.5:4443/readyz": dial tcp 172.17.0.5:4443: connect: connection refused
  Warning  Unhealthy       24h (x7 over 24h)     kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500
  Normal   SandboxChanged  10m                   kubelet            Pod sandbox changed, it will be killed and re-created.
  Warning  Unhealthy       10m (x2 over 10m)     kubelet            Readiness probe failed: Get "https://172.17.0.3:4443/readyz": dial tcp 172.17.0.3:4443: connect: connection refused
  Warning  Unhealthy       10m                   kubelet            Liveness probe failed: Get "https://172.17.0.3:4443/livez": dial tcp 172.17.0.3:4443: connect: connection refused
  Warning  Unhealthy       9m51s (x2 over 10m)   kubelet            Liveness probe failed: Get "https://172.17.0.3:4443/livez": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Normal   Killing         9m51s                 kubelet            Container metrics-server failed liveness probe, will be restarted
  Warning  Unhealthy       9m50s (x4 over 10m)   kubelet            Readiness probe failed: Get "https://172.17.0.3:4443/readyz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Normal   Started         9m49s (x2 over 10m)   kubelet            Started container metrics-server
  Normal   Pulled          9m49s (x2 over 10m)   kubelet            Container image "k8s.gcr.io/metrics-server/metrics-server:v0.6.1@sha256:5ddc6458eb95f5c70bd13fdab90cbd7d6ad1066e5b528ad1dcb28b76c5fb2f00" already present on machine
  Normal   Created         9m49s (x2 over 10m)   kubelet            Created container metrics-server
  Warning  Unhealthy       9m3s (x8 over 9m48s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500

nicoclau avatar Oct 06 '22 16:10 nicoclau

I tried with Minikube and still to no avail :(

 kubectl get --raw /api/v1/nodes/minikube/proxy/stats/summary
{
 "node": {
  "nodeName": "minikube",
  "systemContainers": [
   {
    "name": "kubelet",
    "startTime": "2022-10-06T15:56:01Z",
    "cpu": {
     "time": "2022-10-06T15:56:01Z",
     "usageNanoCores": 0,
     "usageCoreNanoSeconds": 0
    },
    "memory": {
     "time": "2022-10-06T15:56:01Z",
     "usageBytes": 114909184,
     "workingSetBytes": 33673216,
     "rssBytes": 29732864,
     "pageFaults": 7755,
     "majorPageFaults": 528
    }
   },
   {
    "name": "runtime",
    "startTime": "2022-10-06T15:55:49Z",
    "cpu": {
     "time": "2022-10-06T15:56:01Z",
     "usageNanoCores": 0,
     "usageCoreNanoSeconds": 0
    },
    "memory": {
     "time": "2022-10-06T15:56:01Z",
     "usageBytes": 55218176,
     "workingSetBytes": 54542336,
     "rssBytes": 52011008,
     "pageFaults": 32967,
     "majorPageFaults": 0
    }
   },
   {
    "name": "pods",
    "startTime": "2022-10-06T15:56:01Z",
    "cpu": {
     "time": "2022-10-06T16:03:23Z",
     "usageNanoCores": 97916797,
     "usageCoreNanoSeconds": 93816632700
    },
    "memory": {
     "time": "2022-10-06T16:03:23Z",
     "availableBytes": 12773404672,
     "usageBytes": 1022820352,
     "workingSetBytes": 570683392,
     "rssBytes": 471523328,
     "pageFaults": 0,
     "majorPageFaults": 0
    }
   }
  ],
  "startTime": "2022-10-06T15:55:09Z",
  "cpu": {
   "time": "2022-10-06T16:03:23Z",
   "usageNanoCores": 140348378,
   "usageCoreNanoSeconds": 136635717800
  },
  "memory": {
   "time": "2022-10-06T16:03:23Z",
   "availableBytes": 12297957376,
   "usageBytes": 1769955328,
   "workingSetBytes": 1046130688,
   "rssBytes": 702144512,
   "pageFaults": 40491,
   "majorPageFaults": 165
  },
  "network": {
   "time": "2022-10-06T16:03:23Z",
   "name": "eth0",
   "rxBytes": 389705,
   "rxErrors": 0,
   "txBytes": 1904451,
   "txErrors": 0,
   "interfaces": [
    {
     "name": "tunl0",
     "rxBytes": 0,
     "rxErrors": 0,
     "txBytes": 0,
     "txErrors": 0
    },
    {
     "name": "sit0",
     "rxBytes": 0,
     "rxErrors": 0,
     "txBytes": 0,
     "txErrors": 0
    },
    {
     "name": "eth0",
     "rxBytes": 389705,
     "rxErrors": 0,
     "txBytes": 1904451,
     "txErrors": 0
    }
   ]
  },
  "fs": {
   "time": "2022-10-06T16:03:23Z",
   "availableBytes": 239631134720,
   "capacityBytes": 269490393088,
   "usedBytes": 16098586624,
   "inodesFree": 16504036,
   "inodes": 16777216,
   "inodesUsed": 273180
  },
  "runtime": {
   "imageFs": {
    "time": "2022-10-06T16:03:23Z",
    "availableBytes": 239631134720,
    "capacityBytes": 269490393088,
    "usedBytes": 1156495716,
    "inodesFree": 16504036,
    "inodes": 16777216,
    "inodesUsed": 273180
   }
  },
  "rlimit": {
   "time": "2022-10-06T16:03:29Z",
   "maxpid": 4194304,
   "curproc": 967
  }
 },
 "pods": []
}

I had the events:

kubectl describe po metrics-server-7c57dd9c77-7cmwc -n kube-system
Name:                 metrics-server-7c57dd9c77-7cmwc
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 minikube/192.168.49.2
Start Time:           Wed, 05 Oct 2022 17:42:22 +0200
Labels:               k8s-app=metrics-server
                      pod-template-hash=7c57dd9c77
Annotations:          <none>
Status:               Running
IP:                   172.17.0.3
IPs:
  IP:           172.17.0.3
Controlled By:  ReplicaSet/metrics-server-7c57dd9c77
Containers:
  metrics-server:
    Container ID:  docker://b42c61d93d1bbbd603c325d8c7163c9e3ae070206df0e1f938f3bc61169a0674
    Image:         k8s.gcr.io/metrics-server/metrics-server:v0.6.1@sha256:5ddc6458eb95f5c70bd13fdab90cbd7d6ad1066e5b528ad1dcb28b76c5fb2f00
    Image ID:      docker-pullable://k8s.gcr.io/metrics-server/metrics-server@sha256:5ddc6458eb95f5c70bd13fdab90cbd7d6ad1066e5b528ad1dcb28b76c5fb2f00
    Port:          4443/TCP
    Host Port:     0/TCP
    Args:
      --cert-dir=/tmp
      --secure-port=4443
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --kubelet-use-node-status-port
      --metric-resolution=60s
      --kubelet-insecure-tls
    State:          Running
      Started:      Thu, 06 Oct 2022 17:56:36 +0200
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Thu, 06 Oct 2022 17:56:12 +0200
      Finished:     Thu, 06 Oct 2022 17:56:36 +0200
    Ready:          True
    Restart Count:  3
    Requests:
      cpu:        100m
      memory:     200Mi
    Liveness:     http-get https://:https/livez delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get https://:https/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /tmp from tmp-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-k54wr (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  tmp-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-k54wr:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason          Age                   From               Message
  ----     ------          ----                  ----               -------
  Normal   Scheduled       24h                   default-scheduler  Successfully assigned kube-system/metrics-server-7c57dd9c77-7cmwc to minikube
  Normal   Pulling         24h                   kubelet            Pulling image "k8s.gcr.io/metrics-server/metrics-server:v0.6.1@sha256:5ddc6458eb95f5c70bd13fdab90cbd7d6ad1066e5b528ad1dcb28b76c5fb2f00"
  Normal   Pulled          24h                   kubelet            Successfully pulled image "k8s.gcr.io/metrics-server/metrics-server:v0.6.1@sha256:5ddc6458eb95f5c70bd13fdab90cbd7d6ad1066e5b528ad1dcb28b76c5fb2f00" in 16.3926953s
  Normal   Created         24h                   kubelet            Created container metrics-server
  Normal   Started         24h                   kubelet            Started container metrics-server
  Warning  Unhealthy       24h                   kubelet            Readiness probe failed: Get "https://172.17.0.5:4443/readyz": dial tcp 172.17.0.5:4443: connect: connection refused
  Warning  Unhealthy       24h (x8 over 24h)     kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500
  Warning  Unhealthy       24h                   kubelet            Liveness probe failed: Get "https://172.17.0.5:4443/livez": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy       24h                   kubelet            Readiness probe failed: Get "https://172.17.0.5:4443/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Normal   SandboxChanged  24h                   kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled          24h                   kubelet            Container image "k8s.gcr.io/metrics-server/metrics-server:v0.6.1@sha256:5ddc6458eb95f5c70bd13fdab90cbd7d6ad1066e5b528ad1dcb28b76c5fb2f00" already present on machine
  Normal   Created         24h                   kubelet            Created container metrics-server
  Normal   Started         24h                   kubelet            Started container metrics-server
  Warning  Unhealthy       24h                   kubelet            Liveness probe failed: Get "https://172.17.0.5:4443/livez": dial tcp 172.17.0.5:4443: connect: connection refused
  Warning  Unhealthy       24h (x2 over 24h)     kubelet            Readiness probe failed: Get "https://172.17.0.5:4443/readyz": dial tcp 172.17.0.5:4443: connect: connection refused
  Warning  Unhealthy       24h (x7 over 24h)     kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500
  Normal   SandboxChanged  10m                   kubelet            Pod sandbox changed, it will be killed and re-created.
  Warning  Unhealthy       10m (x2 over 10m)     kubelet            Readiness probe failed: Get "https://172.17.0.3:4443/readyz": dial tcp 172.17.0.3:4443: connect: connection refused
  Warning  Unhealthy       10m                   kubelet            Liveness probe failed: Get "https://172.17.0.3:4443/livez": dial tcp 172.17.0.3:4443: connect: connection refused
  Warning  Unhealthy       9m51s (x2 over 10m)   kubelet            Liveness probe failed: Get "https://172.17.0.3:4443/livez": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Normal   Killing         9m51s                 kubelet            Container metrics-server failed liveness probe, will be restarted
  Warning  Unhealthy       9m50s (x4 over 10m)   kubelet            Readiness probe failed: Get "https://172.17.0.3:4443/readyz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Normal   Started         9m49s (x2 over 10m)   kubelet            Started container metrics-server
  Normal   Pulled          9m49s (x2 over 10m)   kubelet            Container image "k8s.gcr.io/metrics-server/metrics-server:v0.6.1@sha256:5ddc6458eb95f5c70bd13fdab90cbd7d6ad1066e5b528ad1dcb28b76c5fb2f00" already present on machine
  Normal   Created         9m49s (x2 over 10m)   kubelet            Created container metrics-server
  Warning  Unhealthy       9m3s (x8 over 9m48s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500

I see. But this is not an issue of metrics-server. It is necessary to analyze why the kubelet's stats/summary endpoint does not get the cpu statistics

yangjunmyfm192085 avatar Oct 08 '22 13:10 yangjunmyfm192085

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Jan 06 '23 13:01 k8s-triage-robot

Not an issue with metrics-server, closing. /close

rexagod avatar Jan 30 '23 09:01 rexagod

@rexagod: Closing this issue.

In response to this:

Not an issue with metrics-server, closing. /close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot avatar Jan 30 '23 09:01 k8s-ci-robot