Missing instructions on how to add k8s cluster

[alikhil]

Please, add detailed instructions on how to add k8s cluster. It's not obvious

Thank you!

[LanDeleih]

Hi. You need to set check box like on image below. In URL section just insert your kube-api url, like "https://192.168.1.1:6443"

In config section, i inserted my ~/.kube/config and ca.crt

Hope it's helpful.

[SergeiSporyshev]

Hi. We plan to add this instructions to readme

[Myafq]

Is there any way to add cluster with token authentication? Or maybe even just using grafana's service account? I mean i don't see any easy way to connect to my GKE cluster right now.

[viberan]

same here, can't use it with a cluster on GKE

[Quicksy]

@LanDeleih Tried to set settings in the same way as on the screenshot. Data got from my ~/.kube/config (certs and key got from there). But I keep getting the error="Failed to parse TLS CA PEM certificate" message. Could you tell me what could be wrong here?

[Quicksy]

Soooo... Okay, I were able to connect cluster - just used CA from apiserver.crt, clientcert and client key from files apiserver-kubelet-client.* I can see info in "cluster" section of this plugin, but the dashboards are empty for some reason.

[Kurmaev]

Soooo... Okay, I were able to connect cluster - just used CA from apiserver.crt, clientcert and client key from files apiserver-kubelet-client.* I can see info in "cluster" section of this plugin, but the dashboards are empty for some reason.

I guess you need to have configured https://github.com/kubernetes/kube-state-metrics (/kubernetes folder contains all manifests you need) , prometheus (probably already configured) and node-exporter (you can take manifests for it from here https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/prometheus . I've just replaced image version with latest 0.18)

[Quicksy]

@Kurmaev prometheus(1.7.0), node-exporter(0.14.0), kube-state-metrics (1.7.2) already configured and metrics are available for other dashboards. The prometheus'es datasource works fine. But here I can see a lot of "Prometheus metrics unavailable" messages.

[Kurmaev]

@Quicksy Ah, yeah, see them too. I had not working:

1. node dashboard - charts
2. nodes overview - table with requested/allocated
3. too many Prometheus metrics unavailable First issue was fixed by adding for prometheus __meta_kubernetes_node_name -> node relabeling Second was partially fixed too, i see numbers here, but it's has bad percents (1.266 GiB (42698.52 % ) ). So i'm on step 2 right now

[Kurmaev]

@Quicksy Ok got it. Table with requested/allocated - issue comes from fact that my setup gives mem info as '3110Mi', when looks like excepted value is '3110000Ki'. Put pullrequest for that too many Prometheus metrics unavailable - this notify is shown when one of requested/used metrics is inavailable, i.e. if you didnt specified mem requests for pod you will see that notification near both mem used/requested

[Quicksy]

@Kurmaev thank you for info :) Could you tell me what and where should I change on this step: First issue was fixed by adding for prometheus __meta_kubernetes_node_name -> node relabeling

[Kurmaev]

@Quicksy prometheus.yml, for all scrape configs:

   ........
    scrape_configs:
    - job_name: 'kubernetes-kubelet'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      .......
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        replacement: '$1'
        target_label: node
    .....

Although i thinks something goes overcalculated: kubectl top nodes reports one value, and dashboard reports value like 3 or more times bigger. I've checked output for /api/v1/nodes/<node-name>/proxy/metrics/cadvisor and looks like it have some aggregated values that lead to increased values. Tried to filter out some values, but aggregated value still not match. For now idk how to fix that. Might be waiting helm charts from authors would be better idea that we have for a moment :)

[Quicksy]

@Kurmaev yeah, i tried this before, but this change broke a lot of data in the dashboards for some reason. Okay, just gonna wait for helm charts from authors :) Thank you for your help!

[feadosiy]

It seems like there is no way to connect plugin to GKE

[bianchi2]

I keep getting:

t=2019-09-25T13:09:51+0000 lvl=eror msg="Unable to load TLS certificate" logger=context userId=1 orgId=1 uname=admin error="tls: private key does not match public key"

though I used crt and key from ~/.kube/config which works fine for kubectl.

[bykvaadm]

why bearer token section is unavailable?

[SergeiSporyshev]

@bykvaadm try our latest release from repo (https://github.com/devopsprodigy/kubegraf/releases/tag/v1.1.1.1) or update plugin version from your settings page

We fixed this bug (on graphana v5.x)

[SergeiSporyshev]

It seems like there is no way to connect plugin to GKE @feadosiy try our latest release)

[trhumphries]

The manifest for Node export for Prometheus no longer exists are you switching to metrics server?

Or maybe I should just use the helm charts for Prometheus Node Exporter and kube-state-metrics

[nugut]

I did everything according to the instructions from the project README.

1. grafana-cli plugins install and restart Grafana server
2. Apply Kubernetes manifests
3. Create a grafana-kubegraf user private key and certificate on master node, get the token.
4. Create cluster in interface
5. Enter settings: Kubernetes API address and port from results of "kubectl cluster-info"
6. Add certificate and key.

But when I press "Save & Test" button, "Bad Request" message appeared. If change authorization on "Access via token" then "Bad Gateway" message appeared.

Grafana is installed as part of prometheus-operator Helm chart. Prometheus plugin, dashboards for Kubernetes etc. work fine.

[SergeiSporyshev]

Hi, @nugut try to enable option "Skip TLS Verification"