kube-ops-view icon indicating copy to clipboard operation
kube-ops-view copied to clipboard

Published 20 hours ago • verified publisher icon hjacobs

heapster metrics not available when running in OpenShift

Open raffaelespazzoli opened this issue 8 years ago • 12 comments

This is probably a newbie question, so apologies. When running ops-view in openshift (pod mode), I get the following errors:

WARNING:kube_ops_view.kubernetes:Failed to query node metrics 172-30-0-1:443: 403 Forbidden
WARNING:kube_ops_view.kubernetes:Failed to query pod metrics for cluster 172-30-0-1:443: 403 Forbidden

looking at the code it seems to be failing at lines kubernetes.py 104 and 115 when executing a query of the type: /api/v1/namespaces/kube-system/services/heapster/proxy/apis/metrics/v1alpha1/nodes

few things don't add up, I gave cluster-admin to the ops-vew pod service account, so nothing should be forbidden. Also I am not familiar with that API path, but in openshift heapster runs in the openshift-infra namespace. So I tried the following:

curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: oc/v1.5.2+43a9be4 (linux/amd64) kubernetes/43a9be4" -H "Authorization: Bearer V83ddLYaBl-i_5QK-q2ndm-Q39ZjM7cNy0ieV7mESPI" https://192.168.99.100:8443/api/v1/namespaces/openshift-infra/services/heapster/proxy/apis/metrics/v1alpha1/pods

which returns

Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 192.168.99.100...
* TCP_NODELAY set
* Connected to 192.168.99.100 (192.168.99.100) port 8443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
* ALPN, server accepted to use http/1.1
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=10.0.2.15
* 	start date: Apr 07 14:29:35 2017 GMT
* 	expire date: Apr 07 14:29:36 2019 GMT
* 	common name: 10.0.2.15
* 	issuer: CN=openshift-signer@1491575375
> GET /api/v1/namespaces/openshift-infra/services/heapster/proxy/apis/metrics/v1alpha1/pods HTTP/1.1
> Host: 192.168.99.100:8443
> Accept: application/json
> User-Agent: oc/v1.5.2+43a9be4 (linux/amd64) kubernetes/43a9be4
> Authorization: Bearer V83ddLYaBl-i_5QK-q2ndm-Q39ZjM7cNy0ieV7mESPI
> 
< HTTP/1.1 503 Service Unavailable
< Cache-Control: no-store
< Date: Thu, 20 Apr 2017 01:50:19 GMT
< Content-Length: 129
< Content-Type: text/plain; charset=utf-8
< 
Error: 'malformed HTTP response "\x15\x03\x01\x00\x02\x02"'
* Curl_http_done: called premature == 0
* Connection #0 to host 192.168.99.100 left intact
Trying to reach: 'http://172.17.0.10:8082/apis/metrics/v1alpha1/pods'

The last line makes me think that this would be the right path, yet the service still fails. Could it be because of the v1alpha1? Is that a new API? In which version of heapster was it introduced?

If I try to run

curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: oc/v1.5.2+43a9be4 (linux/amd64) kubernetes/43a9be4" -H "Authorization: Bearer V83ddLYaBl-i_5QK-q2ndm-Q39ZjM7cNy0ieV7mESPI" https://192.168.99.100:8443/api/v1/namespaces/kube-system/services/heapster/proxy/apis/metrics/v1alpha1/pods

I get a 404.

if I try to run ops-vew in proxy mode with a cluster admin, I get

WARNING:kube_ops_view.kubernetes:Failed to query node metrics localhost:8001: 404 Not Found
WARNING:kube_ops_view.kubernetes:Failed to query pod metrics for cluster localhost:8001: 404 Not Found

so again the heapster calls are failing and this time with a 404.

I'm not sure if I'm doing anything wrong here, please advise.

Also, would it be possible to detect if we are running in a openshift cluster and change the path of the heapster calls?

raffaelespazzoli avatar Apr 20 '17 02:04 raffaelespazzoli

@raffaelespazzoli can you try kubectl top nodes --v=9 and tell me the URL/path to Heapster it's logging?

hjacobs avatar Apr 20 '17 07:04 hjacobs

translating that command in OpenShift parlance here is what I had to do to make it work:

oc --loglevel=9 adm top node --heapster-namespace=openshift-infra --heapster-scheme=https

which returned:

I0420 07:26:01.266336    8014 loader.go:354] Config loaded from file /home/rspazzol/.kube/config
I0420 07:26:01.270788    8014 cached_discovery.go:112] returning cached discovery info from /home/rspazzol/.kube/192.168.99.100_8443/servergroups.json
I0420 07:26:01.271735    8014 round_trippers.go:299] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: oc/v1.5.2+43a9be4 (linux/amd64) kubernetes/43a9be4" -H "Authorization: Bearer V83ddLYaBl-i_5QK-q2ndm-Q39ZjM7cNy0ieV7mESPI" https://192.168.99.100:8443/api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/apis/metrics/v1alpha1/nodes?labelSelector=
I0420 07:26:01.364415    8014 round_trippers.go:318] GET https://192.168.99.100:8443/api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/apis/metrics/v1alpha1/nodes?labelSelector= 200 OK in 92 milliseconds
I0420 07:26:01.364445    8014 round_trippers.go:324] Response Headers:
I0420 07:26:01.364458    8014 round_trippers.go:327]     Cache-Control: no-store
I0420 07:26:01.364467    8014 round_trippers.go:327]     Content-Type: application/json
I0420 07:26:01.364475    8014 round_trippers.go:327]     Date: Thu, 20 Apr 2017 11:26:05 GMT
I0420 07:26:01.364995    8014 round_trippers.go:299] curl -k -v -XGET  -H "Authorization: Bearer V83ddLYaBl-i_5QK-q2ndm-Q39ZjM7cNy0ieV7mESPI" -H "Accept: application/json, */*" -H "User-Agent: oc/v1.5.2+43a9be4 (linux/amd64) kubernetes/43a9be4" https://192.168.99.100:8443/api/v1/nodes
I0420 07:26:01.376321    8014 round_trippers.go:318] GET https://192.168.99.100:8443/api/v1/nodes 200 OK in 11 milliseconds
I0420 07:26:01.376363    8014 round_trippers.go:324] Response Headers:
I0420 07:26:01.376374    8014 round_trippers.go:327]     Cache-Control: no-store
I0420 07:26:01.376383    8014 round_trippers.go:327]     Content-Type: application/json
I0420 07:26:01.376391    8014 round_trippers.go:327]     Date: Thu, 20 Apr 2017 11:26:05 GMT
I0420 07:26:01.376843    8014 request.go:905] Response Body: {"kind":"NodeList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/nodes","resourceVersion":"82599"},"items":[{"metadata":{"name":"192.168.99.100","selfLink":"/api/v1/nodes/192.168.99.100","uid":"abca3cb4-1b9e-11e7-8c87-0800270667c1","resourceVersion":"82598","creationTimestamp":"2017-04-07T14:29:55Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/hostname":"192.168.99.100"}},"spec":{"externalID":"192.168.99.100"},"status":{"capacity":{"alpha.kubernetes.io/nvidia-gpu":"0","cpu":"2","memory":"12304368Ki","pods":"20"},"allocatable":{"alpha.kubernetes.io/nvidia-gpu":"0","cpu":"2","memory":"12304368Ki","pods":"20"},"conditions":[{"type":"OutOfDisk","status":"False","lastHeartbeatTime":"2017-04-20T11:25:55Z","lastTransitionTime":"2017-04-19T21:22:41Z","reason":"KubeletHasSufficientDisk","message":"kubelet has sufficient disk space available"},{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2017-04-20T11:25:55Z","lastTransitionTime":"2017-04-07T14:29:55Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2017-04-20T11:25:55Z","lastTransitionTime":"2017-04-07T14:29:55Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"Ready","status":"True","lastHeartbeatTime":"2017-04-20T11:25:55Z","lastTransitionTime":"2017-04-20T11:10:29Z","reason":"KubeletReady","message":"kubelet is posting ready status"}],"addresses":[{"type":"LegacyHostIP","address":"192.168.99.100"},{"type":"InternalIP","address":"192.168.99.100"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"66637ea6e73641618287fcaae5b1c801","systemUUID":"329C578E-DD92-4CC0-A961-2210C24A2FA4","bootID":"1f5b7542-02aa-4f27-bc24-03b481480412","kernelVersion":"3.10.0-514.6.2.el7.x86_64","osImage":"Red Hat Enterprise Linux Server 7.3 (Maipo)","containerRuntimeVersion":"docker://1.12.5","kubeletVersion":"v1.4.0+776c994","kubeProxyVersion":"v1.4.0+776c994","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["registry.access.redhat.com/openshift3/metrics-hawkular-metrics@sha256:2a319bbb1e88ecb0d94d845ae7ef19d8cb83815a5ec7fee0a704366afa39a670","registry.access.redhat.com/openshift3/metrics-hawkular-metrics:3.4.0"],"sizeBytes":1508059267},{"names":["registry.access.redhat.com/openshift3/metrics-hawkular-metrics@sha256:6cac723e050fa19e4e751273c1b961d1b3ac4dcdfdd6d1e9a2dc27294088915f","registry.access.redhat.com/openshift3/metrics-hawkular-metrics:v3.4"],"sizeBytes":1260472748},{"names":["registry.access.redhat.com/openshift3/ose-docker-registry@sha256:4fccc5df60444c3fb8c322661e55718f9f1f398fccf9b956aa999d1a901eca70","registry.access.redhat.com/openshift3/ose-docker-registry:v3.4.1.2"],"sizeBytes":775815912},{"names":["registry.access.redhat.com/openshift3/metrics-deployer@sha256:b0589b2f188390f7a97fb409cb518a3881a37d1d1f406c738d5da31df20c7756","registry.access.redhat.com/openshift3/metrics-deployer:3.4.0"],"sizeBytes":771550491},{"names":["172.30.1.1:5000/ci/jenkins-blueocean@sha256:f1b8cf8efa3274d945da08a65524fc90ddbefd7e99d43d14c85b27a6ca91fb68","172.30.1.1:5000/ci/jenkins-blueocean:latest"],"sizeBytes":730392568},{"names":["registry.access.redhat.com/openshift3/ose-haproxy-router@sha256:8ccf8058b30fc053b094ba955522ea499c691e63f10f1247134b3a51f8a4b98d","registry.access.redhat.com/openshift3/ose-haproxy-router:v3.4.1.2"],"sizeBytes":714559491},{"names":["registry.access.redhat.com/openshift3/ose-deployer@sha256:37adf782e29f09c815ae0bd91299e99ae84e2849b25de100c6581df36c6a7920","registry.access.redhat.com/openshift3/ose-deployer:v3.4.1.2"],"sizeBytes":696008933},{"names":["registry.access.redhat.com/openshift3/ose-sti-builder@sha256:802c025dd4674507247eea73e46bd74e50724f1371d90f4b2220d87c2e35faba","registry.access.redhat.com/openshift3/ose-sti-builder:v3.4.1.2"],"sizeBytes":696008700},{"names":["registry.access.redhat.com/openshift3/ose@sha256:0501560588cceefe72ea1c53a02b75ba20404b5f68ab2ea0755ffe2d51ed63f3","registry.access.redhat.com/openshift3/ose:v3.4.1.2"],"sizeBytes":696007448},{"names":["registry.access.redhat.com/openshift3/jenkins-2-rhel7@sha256:66eea68ebc701a13527a4172feb2c4c0c9acda09efaa59821393498121d97ae0"],"sizeBytes":678497590},{"names":["registry.access.redhat.com/openshift3/metrics-cassandra@sha256:52d66ecc1279dfbead7fd7560315eb097c1141bab282f76142074e0c6f3febb9","registry.access.redhat.com/openshift3/metrics-cassandra:3.4.0"],"sizeBytes":649214546},{"names":["registry.access.redhat.com/openshift3/metrics-cassandra@sha256:dc65fcae16c0c5b8d016eef33e89cdcf8c9fadadf63a8999125348a58dce827e","registry.access.redhat.com/openshift3/metrics-cassandra:v3.4"],"sizeBytes":539330010},{"names":["registry.access.redhat.com/openshift3/metrics-heapster@sha256:e9c5daa896b0a192ade46be34950b84689bb3fc49e42ea2944f44cae69d05609","registry.access.redhat.com/openshift3/metrics-heapster:v3.4"],"sizeBytes":317832749},{"names":["registry.access.redhat.com/openshift3/metrics-heapster@sha256:1b0f9103e7e938e6c3ff91f8b378a24dc38af6ea1ca37068d0a68f69d41d1dd7","registry.access.redhat.com/openshift3/metrics-heapster:3.4.0"],"sizeBytes":317811627},{"names":["docker.io/wongma7/nfs-provisioner@sha256:e51b3ef12530d1b5a23d7329aba14e360ce8fa11a827a10d585a0f482606b1ff","docker.io/wongma7/nfs-provisioner:latest"],"sizeBytes":314893150},{"names":["registry.access.redhat.com/openshift3/ose-pod@sha256:0be97a928657b3a0248ebccc07324fb544c0f54ac40bec928fecb788c309727a","registry.access.redhat.com/openshift3/ose-pod:v3.4.1.2"],"sizeBytes":204973351},{"names":["docker.io/raffaelespazzoli/credscontroller@sha256:220557df6bb42d2f3b980a3fdedb0d670634b69f1b0816b26d6e462bd4de3789","docker.io/raffaelespazzoli/credscontroller:0.0.1"],"sizeBytes":124733026},{"names":["docker.io/hjacobs/kube-ops-view@sha256:7e53625c370e705392b82316e976e2e08ec6d37952b1d03f2cad30c9cc0d2dab","docker.io/hjacobs/kube-ops-view:latest"],"sizeBytes":94250680},{"names":["docker.io/prom/prometheus@sha256:e049c086e35c0426389cd2450ef193f6c18b3d0065b97e5f203fdb254716fa1c","docker.io/prom/prometheus:v1.5.2"],"sizeBytes":79601765},{"names":["docker.io/prom/prometheus@sha256:f4584467a510ca355738727423ce47698290258cfc1feac7dbab120c6f9898b0","docker.io/prom/prometheus:latest"],"sizeBytes":75286574},{"names":["docker.io/vault@sha256:d94bef21674d739961ad45c350a61da30a5f577b709b80270d6da6f9aa209c2a","docker.io/vault:0.6.5"],"sizeBytes":63674810},{"names":["docker.io/redis@sha256:9cd405cd1ec1410eaab064a1383d0d8854d1eef74a54e1e4a92fb4ec7bdc3ee7","docker.io/redis:3.2-alpine"],"sizeBytes":19816272}]}}]}
NAME             CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%   
192.168.99.100   1007m        50%       5896Mi          49%

raffaelespazzoli avatar Apr 20 '17 11:04 raffaelespazzoli

so the right url to call when in openshift is: api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/apis/metrics/v1alpha1 I'm wondering if you guys could support this.

Thanks Raffaele

raffaelespazzoli avatar Apr 24 '17 00:04 raffaelespazzoli

@raffaelespazzoli yeah, I will check that and make sure it works with OpenShift (I never tried it out before..).

hjacobs avatar Apr 24 '17 06:04 hjacobs

@raffaelespazzoli what OpenShift version are you running?

hjacobs avatar Apr 24 '17 06:04 hjacobs

3.5.

On Apr 24, 2017 2:42 AM, "Henning Jacobs" [email protected] wrote:

@raffaelespazzoli https://github.com/raffaelespazzoli what OpenShift version are you running?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hjacobs/kube-ops-view/issues/137#issuecomment-296541731, or mute the thread https://github.com/notifications/unsubscribe-auth/AF5I3Cgp4b3JUr6SS7h8qH0hI9JMkln9ks5rzERygaJpZM4NCc1Q .

raffaelespazzoli avatar Apr 24 '17 10:04 raffaelespazzoli

running ocp 3.5 over here, not sure why but getting the 403 forbidden error.

i found the URL in the kubernetes.py file as you described above and it looks correct. what should i try next please @raffaelespazzoli ?

ironicbadger avatar Aug 02 '17 15:08 ironicbadger

I have cloned this repo and ported it to openshift here: https://github.com/raffaelespazzoli/kube-ops-view , I'd be happy to create a PR with the necessary change to make openshift work if someone can assist (meaning right now I have just assumed we are running on openshift, the right approach would be to have a configuration setting or self detect on which platform we are running).

@IronicBadger please refer to my repo, for the 403 you probably are missing some permissions, try running as cluster admin at first.

raffaelespazzoli avatar Aug 02 '17 15:08 raffaelespazzoli

@raffaelespazzoli can you provide me with some "simple" instructions on how to run OpenShift locally including Heapster? I tried out minishift, but I did not succeed with installing metrics etc....

hjacobs avatar Aug 02 '17 15:08 hjacobs

For our clusters we have moved heapster out of kube-system into our own tools namespace. This heapster path should be provided as a parameter we can override.

Here's how we tell kube dashboard where heapster is.

  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: kubernetes-dashboard
    spec:
      containers:
      - args:
        - --auto-generate-certificates
        - --heapster-host=http://heapster.kube-tools.svc.cluster.local:8082
        image: k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.2
        imagePullPolicy: IfNotPresent

bobhenkel avatar Feb 16 '18 22:02 bobhenkel

Hi @raffaelespazzoli and @hjacobs

! I've tried @raffaelespazzoli's fork with the OCP code to use OpenShift's oAuth server and deployed on OpenShift 3.7. Apparently, it doesn't work when the router uses a self-signed certificate:

[2018-08-06 22:59:24,855] ERROR in app: Exception on /login/authorized [GET]
--
  | Traceback (most recent call last):
  | File "/usr/lib/python3.6/urllib/request.py", line 1318, in do_open
  | encode_chunked=req.has_header('Transfer-encoding'))
  | File "/usr/lib/python3.6/http/client.py", line 1239, in request
  | self._send_request(method, url, body, headers, encode_chunked)
  | File "/usr/lib/python3.6/http/client.py", line 1285, in _send_request
  | self.endheaders(body, encode_chunked=encode_chunked)
  | File "/usr/lib/python3.6/http/client.py", line 1234, in endheaders
  | self._send_output(message_body, encode_chunked=encode_chunked)
  | File "/usr/lib/python3.6/http/client.py", line 1026, in _send_output
  | self.send(msg)
  | File "/usr/lib/python3.6/http/client.py", line 964, in send
  | self.connect()
  | File "/usr/lib/python3.6/http/client.py", line 1400, in connect
  | server_hostname=server_hostname)
  | File "/usr/lib/python3.6/site-packages/gevent/_ssl3.py", line 60, in wrap_socket
  | _session=session)
  | File "/usr/lib/python3.6/site-packages/gevent/_ssl3.py", line 232, in __init__
  | raise x
  | File "/usr/lib/python3.6/site-packages/gevent/_ssl3.py", line 228, in __init__
  | self.do_handshake()
  | File "/usr/lib/python3.6/site-packages/gevent/_ssl3.py", line 545, in do_handshake
  | self._sslobj.do_handshake()
  | ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)

Would that be a kube-ops-view bug or something specific to the OCP fork?

slaterx avatar Aug 06 '18 23:08 slaterx

I just tried Google Kubernetes Engine nad I can't see current usage metrics from cluster, but kubectl top node shows them.

morsik@alderaan ~ $ kubectl top node
NAME                           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
gke-ohc-medium-bcd29d5b-dbzk   30m          1%     353Mi           12%
gke-ohc-small-dfb3afc9-77vj    18m          1%     336Mi           29%
gke-ohc-small-dfb3afc9-9kfc    40m          4%     705Mi           60%
gke-ohc-small-dfb3afc9-v0dg    18m          1%     342Mi           29%

morsik@alderaan ~ $ kubectl top pod
NAME                                   CPU(cores)   MEMORY(bytes)
kube-ops-view-57dc7b767b-lqgl8         5m           47Mi
kube-ops-view-redis-74578dcc5d-jtsrv   1m           3Mi

morsik@alderaan ~$ oc logs kube-ops-view-57dc7b767b-lqgl8 | tail
WARNING:kube_ops_view.kubernetes:Failed to query node metrics 10-39-240-1:443: 404 Not Found
WARNING:kube_ops_view.kubernetes:Failed to query pod metrics for cluster 10-39-240-1:443: 404 Not Found
10.166.0.4 - - [2018-11-08 19:56:48] "GET /health HTTP/1.1" 200 117 0.000744
10.166.0.4 - - [2018-11-08 19:56:48] "GET /health HTTP/1.1" 200 117 0.041777
10.166.0.4 - - [2018-11-08 19:56:49] "GET /health HTTP/1.1" 200 117 0.000664
WARNING:kube_ops_view.kubernetes:Failed to query node metrics 10-39-240-1:443: 404 Not Found
WARNING:kube_ops_view.kubernetes:Failed to query pod metrics for cluster 10-39-240-1:443: 404 Not Found
10.36.0.1 - - [2018-11-08 19:56:54] "GET /health HTTP/1.1" 200 117 0.000697
WARNING:kube_ops_view.kubernetes:Failed to query node metrics 10-39-240-1:443: 404 Not Found
WARNING:kube_ops_view.kubernetes:Failed to query pod metrics for cluster 10-39-240-1:443: 404 Not Found

It uses:

morsik@alderaan ~$ kubectl -v=10 top node 2>&1 | grep '] GET '
I1108 20:59:31.942352   62457 round_trippers.go:405] GET https://1.2.3.4/api?timeout=32s 200 OK in 393 milliseconds
I1108 20:59:32.026205   62457 round_trippers.go:405] GET https://1.2.3.4/apis?timeout=32s 200 OK in 83 milliseconds
I1108 20:59:32.233636   62457 round_trippers.go:405] GET https://1.2.3.4/apis/metrics.k8s.io/v1beta1/nodes 200 OK in 202 milliseconds
I1108 20:59:32.353909   62457 round_trippers.go:405] GET https://1.2.3.4/api/v1/nodes 200 OK in 116 milliseconds

Looks like some kind of autodiscovery? This URL is very different than what kube-ops-view uses and what OpenShift uses.

/apis/metrics.k8s.io/v1beta1/nodes contains this:

{
  "kind": "NodeMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes"
  },
  "items": [
    {
      "metadata": {
        "name": "gke-ohc-medium-bcd29d5b-dbzk",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/gke-ohc-medium-bcd29d5b-dbzk",
        "creationTimestamp": "2018-11-08T20:01:15Z"
      },
      "timestamp": "2018-11-08T20:01:00Z",
      "window": "1m0s",
      "usage": {
        "cpu": "28m",
        "memory": "339844Ki"
      }
    },
    {
      "metadata": {
        "name": "gke-ohc-small-dfb3afc9-9kfc",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/gke-ohc-small-dfb3afc9-9kfc",
        "creationTimestamp": "2018-11-08T20:01:15Z"
      },
      "timestamp": "2018-11-08T20:01:00Z",
      "window": "1m0s",
      "usage": {
        "cpu": "41m",
        "memory": "723352Ki"
      }
    },
    {
      "metadata": {
        "name": "gke-ohc-small-dfb3afc9-v0dg",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/gke-ohc-small-dfb3afc9-v0dg",
        "creationTimestamp": "2018-11-08T20:01:15Z"
      },
      "timestamp": "2018-11-08T20:01:00Z",
      "window": "1m0s",
      "usage": {
        "cpu": "18m",
        "memory": "338696Ki"
      }
    },
    {
      "metadata": {
        "name": "gke-ohc-small-dfb3afc9-77vj",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/gke-ohc-small-dfb3afc9-77vj",
        "creationTimestamp": "2018-11-08T20:01:15Z"
      },
      "timestamp": "2018-11-08T20:01:00Z",
      "window": "1m0s",
      "usage": {
        "cpu": "18m",
        "memory": "339348Ki"
      }
    }
  ]
}

morsik avatar Nov 08 '18 20:11 morsik