Issue with GKE cluster

[pvikas]

Hi,

I have a standalone grafana server with the plugin installed. Was able to successfully connect to the GKE cluster (the node-exporter and kube-state-metrics) were also successfully deployed automatically. I went ahead and deployed Prometheus in the cluster and updated the scrape config to match the one provided in the add-cluster page.

Thought I can see the node list, services, namespaces and PODs, none of the metrics are shown. All I get in the dashboard is NA and an error stating 'Datasource named $datasource was not found'.

Please help.

Thanks, Vikas

[pvikas]

This was a honest newbie mistake.

Steps to be followed:

• Install grafana on a VM
• Install the plugin
• Setup prometheus using HELM in GKE cluster
• Update the scrape config with the one suggested in this plugin's create dashboard page
• Expose the prometheus endpoint
• Create a datasource pointing to the exposed prometheus endpoint
• Get the credentials (username/password) from GKE cluster info page, also the master URL
• Create a new cluster, select the Basic auth option and use the above username/password and master
• Select the prometheus datasoruce created earlier
• Save

Regards, Vikas

[gabrielfsousa]

@pvikas what prometheus did you install ?

[xmariopereira]

@pvikas only with username / password?

[gabrielfsousa]

i have an independent kubernetes and used TLS Client Auth With CA Cert . and manage to connect.

now my prometheus dont see the kube-state-metrics, any one the give me some guidelines ?

[pvikas]

@gabrielfsousa - Prometheus installed via helm (took the latest stable version) As for no metrics - first guess is check the scrape config for Prometheus

@mariopoeta - this is a setup with VPC in-place. Hence used just the username/password.

[gabrielfsousa]

@pvikas thanks, i'm working with my independent kubernetes cluster

i did my own Prometheus installation and works. now all metrics are working, but did somethings difference

kube-state-metrics is from https://github.com/kubernetes/kube-state-metrics and the prometheus.yml is this

scrape_configs:
    - job_name: 'kubernetes-kubelet'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics
    - job_name: 'kubernetes-cadvisor'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
    - job_name: 'kubernetes-kube-state'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name
      - source_labels: [__meta_kubernetes_pod_label_grafanak8sapp]
        regex: .*true.*
        action: keep
      - source_labels: ['__meta_kubernetes_pod_label_daemon', '__meta_kubernetes_pod_node_name']
        regex: 'node-exporter;(.*)'
        action: replace
        target_label: nodename
    - job_name: 'kubernetes-pods'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name

whit out kubernetes-pods i dont have the kube_ metrics

[cmorent]

Hello there, I am struggling with the same kind of issue...

I have a prometheus environment running within my cluster, installed using the stable helm chart, working fine for other dashboards and metrics collection.

I deployed grafana using the stable helm chart too and I'm struggling with the authentication part. I use the cluster endpoint ip as URL like: https://<endpoint> along with the username, password and cluster CA certificate available in the GKE cluster details. I am also checking the Basic Auth, With CA Cert and Skip TLS verification options.

If I click on the save button, it says that everything went fine and was successfully able to connect to the cluster. However, if I hit the deploy button, it says that the datasource was added but that it was not able to connect to the cluster...

Am I doing something wrong?

[josepedro]

I have the same problem with GKE stuffs... =(

[ghost]

it is working on GKE and on separate Kube cluster (on VMs)

1. Install via Kube cluster
2. Install via Helm prometheus and grafana
3. Install kube plugin on grafana
4. Configure datasource
5. Done.

If you have any additional questions please let me know and I will try to help you.

I have just one problem which related to Node Cluster data on dashboard.

[neodani]

it is working on GKE and on separate Kube cluster (on VMs)

1. Install via Kube cluster

2. Install via Helm prometheus and grafana

3. Install kube plugin on grafana

4. Configure datasource

5. Done.

If you have any additional questions please let me know and I will try to help you.

I have just one problem which related to Node Cluster data on dashboard.

Are you still using the plugin in your GKE cluster?

What steps did you follow to connect the plugin with your cluster?

I used prometheus-operator helm

Thank you very much