Grafana dashboard not showing gluster volumes

[JohnStrunk]

During testing for GCS 1.0-pre, the Grafana dashboard is not displaying metrics for gluster volumes.

Once GCS is deployed, I created the following workload in namespace test:

$ kubectl --kubeconfig=kubeconfig apply -n test -f scratch/test-glusterfs.yml

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-glusterfs
spec:
  storageClassName: glusterfs-csi
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

---

apiVersion: batch/v1
kind: Job
metadata:
  name: fs-performance
spec:
  template:
    metadata:
      name: fs-performance
    spec:
      containers:
        - name: fio
          image: quay.io/johnstrunk/fs-performance:latest
          args: ["clone", "kernel"]
          env:
            - name: TARGET_CAPACITY_MB
              value: "500"
            - name: TARGET_PATH
              value: "/target"
            - name: CLONE_REPO
              value: "https://github.com/gluster/glusterfs.git"
          volumeMounts:
            - name: target
              mountPath: /target
      restartPolicy: Never
      volumes:
        - name: target
          persistentVolumeClaim:
            claimName: test-glusterfs

After the workload started, I viewed the gluster Grafana dashboard, and the "Volume" dropdown only had the entry "None". All graphs either displayed "N/A" or "no data points". Datasource was the default: "Prometheus"

[JohnStrunk]

ping: @sidharthanup @cloudbehl

[umangachapagain]

AFAIK this happens because of multiple port forwarding and stuff that happens from pod to inside cluster to outside cluster, all they way up to the host. Which is why maybe Grafana is not able to resolve it initially. @sidharthanup Correct me if I'm wrong. However, If you create a new Datasource with exact same config from inside the Grafana UI, it works instantly. (This could be documented as a manual fix maybe??) @cloudbehl Any thoughts on this?

[cloudbehl]

@umangachapagain no clues why it is not working now? @sidharthanup any updates from your side?

[sidharthanup]

@aravindavk @aruniiird @shtripat ~~Problem seems to be in the gluster-prometheus side..Once GCS is deployed, gluster-prometheus is not exposing any of the gluster metrics, not even the ps ones. One of the latest refactoring PRs must have prevented this. Looking into it.~~

[sidharthanup]

Disregard the previous comment. The metrics are available and exported properly...GCS was deployed incorrectly and hence the above issue. Now I did some debugging and as @JohnStrunk reports the grafana dashboard does not show any metrics:

This is the volume capacity utilization metrics:

It seems like volume='$Volume' is where the issue lies. Removing it displays the correct metrics:

Also specifying volume=<name of gluster volume> yields the correct metrics( as it should):

So it seems like the Volume variable is not being set by gluster mixin. Not just for this metric but the same issue for others as well. @cloudbehl @umangachapagain Could you guys check it out in the gluster mixins side?

[umangachapagain]

@anmolsachan Can you verify this? As far as I remember, this wasn't the case when we tested against 0.5.

[anmolsachan]

@umangachapagain @sidharthanup Ack, Will verify what's the issue here.

[anmolsachan]

@sidharthanup i deployed the GCS setup, I cannot see any gluster_volume* metrics in in prometheus. The grafana dashboards need those metrics for displaying the info.

It seems like volume='$Volume' is where the issue lies. Removing it displays the correct metrics:

$Volume is intended and needs to be there to select different volumes.

Also specifying volume=<name of gluster volume> yields the correct metrics( as it should):

Thats translates to volume='$Volume where $Volume is the selector.

[JohnStrunk]

Are there any updates or associated issues/PRs?