[kube-prometheus-stack] Grafana Prometheus timeInterval

[dro-sh]

Describe the bug a clear and concise description of what the bug is.

At Grafana Scrape Interval of provisioning Prometheus Datasource is not changing then set another scrapeInterval at values.yaml. Grafana persists data via PVC.

What's your helm version?

version.BuildInfo{Version:"v3.8.0", GitCommit:"d14138609b01886f544b2025f5000351c9eb092e", GitTreeState:"clean", GoVersion:"go1.17.6"}

What's your kubectl version?

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.3", GitCommit:"816c97ab8cff8a1c72eccca1026f7820e93e0d25", GitTreeState:"clean", BuildDate:"2022-01-25T21:17:57Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/arm64"} Server Version: version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.2", GitCommit:"89a4ea3e1e4ddd7f7572286090359983e0387b2f", GitTreeState:"clean", BuildDate:"2023-09-13T09:29:07Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}

Which chart?

kube-prometheus-stack

What's the chart version?

56.9.0

What happened?

1. I deploy kube-prometheus-stack with grafana. Grafana using PVC for storing data. By default timeInterval was set to 30s
2. I change default prometheus scrape interval using scrapeInterval. I connect to pod, check /etc/grafana/provisioning/datasources/datasource.yaml, and at this file timeInverval equals 10s. But when I open Prometheus datasource at grafana UI I found that scrape interval equals 30s. I check DB at PV and at table data_source at column json_data prometheus has timeInterval equals to 30s. How to automatically change value at database?

What you expected to happen?

At Grafana Scrape Interval of provisioning Prometheus Datasource is changing then set another scrapeInterval at values.yaml. Grafana persists data via PVC.

How to reproduce it?

1. Deploy kube-prometheus-stack, setup for grafana PVC.
2. Set scrapeInterval to 10s
3. At Grafana Scrape Interval of provisioning Prometheus Datasource is not changed.

Enter the changed values of values.yaml?

No response

Enter the command that you execute and failing/misfunctioning.

helm upgrade --wait --timeout 360s
--install kube-prometheus-stack prometheus-community/kube-prometheus-stack
--values "${PWD}/kube-prometheus-stack/values.yaml"
--values "${PWD}/kube-prometheus-stack/extra-values.yaml"
--namespace infra --create-namespace
--version 56.9.0

Anything else we need to know?

No response

[zeritti]

Jut checked the reported behaviour. Setting prometheus.prometheusSpec.scrapeInterval to a non-default value gets correctly propagated in the default Prometheus datasource (keeping the rest of the settings at their defaults).

Datasource provisioning from configmaps will only work if the grafana-sc-datasources sidecar has been enabled (default is true).

The sidecar has to successfuly write the file /etc/grafana/provisioning/datasources/datasource.yaml and successfully request a reload at grafana's /api/admin/provisioning/datasources/reload (default set at sidecar.datasources.reloadURL in the grafana chart). As you write, the former succeeded but the latter probably not. Reviewing the sidecar's log messages should provide more info. Some changes have impact on the sidecar's ability to request a reload.