Fix Grafana dashboards for Prometheus Overview

[poornima-krishnasamy]

Background

There are few dashboards which comes out of the box from kube-prometheus-stack. https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14

But some of the dashboard doesnt show in Grafana. e.g. Prometheus Overview https://grafana.live.cloud-platform.service.justice.gov.uk/d/cd596f85-461f-4236-b93a-3574d6f9f2ed/prometheus-overview?orgId=1&refresh=60s

The dashboard is essential to get useful insights about prometheus and debugging purposes.

If the default chart doesnt have a fix, find a alternative grafana dashboard available in open community, import the dashboard for us to use it

Investigate why this is not showing up. Is it due to the angular_js support issue. https://github.com/prometheus-community/helm-charts/issues/4398

Which part of the user docs does this impact

Communicate changes

• [ ] post for #cloud-platform-update
• [ ] Weeknotes item
• [ ] Show the Thing/P&A All Hands/User CoP
• [ ] Announcements channel

Questions / Assumptions

Definition of done

• [ ] Prometheus Overview dashboard fixed
• [ ] readme has been updated
• [ ] user docs have been updated
• [ ] another team member has reviewed
• [ ] smoke tests are green
• [ ] prepare demo for the team

Reference

How to write good user stories

[sj-williams]

The no data issue we are experiencing is due to a change applied to the Prometheus Overview default dashboard config, which now includes a cluster label for the dashboard queries. See this open issue for info.

At this time overriding / disabling the default dashboards via community chart is not available, so for now at least we can apply our own fixed version of the Prometheus Overview dashboard as a custom CM in monitoring repo.

[sj-williams]

Overview dashboard now working https://grafana.live.cloud-platform.service.justice.gov.uk/d/BNLlmx9nk/prometheus-overview-working?orgId=1&refresh=60s