Collect metrics from grafana instance

[paulfantom]

What is missing?

Tobs is not setting up ServiceMonitor for grafana and thus prometheus is not collecting metrics from grafana instance.

Why do we need it?

To increase observability of our stack.

Anything else we need to know?:

Feature can be enabled by enabling ServiceMonitor setting in grafana helm chart - https://github.com/grafana/helm-charts/blob/6578497320d3c4672bab3a3c7fd38dffba1c9aba/charts/grafana/values.yaml#L162-L174. The main issue to consider is that prometheus shipped with kube-prometheus-stack helm chart, by default requires service monitors to have release: XYZ label.