Loading grafana config whenever updated

[avinashdhinwa]

What is missing?

Whenever config object for grafana is updated, grafana should load new changes.

Why do we need it?

On update of grafana config object in k8s, grafana should also get updated with new config so as to avoid manual intervention.

Environment

All

• kube-prometheus version: release-0.6

  ea7a834

Anything else we need to know?:

[paulfantom]

This is possible to implement but not trivial. As far as I can see there are two ways:

1. Automate recreation of grafana pod on config change
2. Use grafana admin API and a sidecar container

Personally, I think using API is a much cleaner way and we can use https://github.com/jimmidyson/configmap-reload but we need to handle authentication to admin endpoints.

We are open to external contributors creating PR with implementation, but we probably won't be actively trying to implement it right now.

[avinashdhinwa]

@paulfantom Cool, I can pick this up. Looks like you have provided enough info, is there anything else I need know before picking this up.

[paulfantom]

Few things:

• everything grafana-related should be in https://github.com/prometheus-operator/kube-prometheus/blob/main/jsonnet/kube-prometheus/components/grafana.libsonnet
• grafana configuration is optional (as seen in https://github.com/prometheus-operator/kube-prometheus/blob/main/jsonnet/kube-prometheus/components/grafana.libsonnet#L69), so reloader sidecar should be too.

I would suggest experimenting with config reloader sidecar handling only reloading of main grafana config. Next, we can figure out if reloading of dashboards and datasources is needed.

[avinashdhinwa]

Thanks @paulfantom for the info, I'll pick this up in coming weeks and will post any queries when I start working on this.

[mrkwtz]

Few things:

• everything grafana-related should be in https://github.com/prometheus-operator/kube-prometheus/blob/main/jsonnet/kube-prometheus/components/grafana.libsonnet
• grafana configuration is optional (as seen in https://github.com/prometheus-operator/kube-prometheus/blob/main/jsonnet/kube-prometheus/components/grafana.libsonnet#L69), so reloader sidecar should be too.

I would suggest experimenting with config reloader sidecar handling only reloading of main grafana config. Next, we can figure out if reloading of dashboards and datasources is needed.

So I dont know if someone is working on it but for us a scenario where we need to reload datasources is when using vault for the datasource's credentials. So we template the datasource configuration with vault and would like to use dynamic database credentials. This requires the datasource.yaml to be reloaded on changes though. For now we are stuck with static credentials that get loaded on startup.

[B1ue-W01f]

This is pretty critical for the above reason and others, there is limited value in configuration through manifests if the manifests are applied but config changes not picked up by the app.