Some Grafana charts not displaying data when duration parameter smaller than scrape interval

[alpeb]

What problem are you trying to solve?

Under some configs, Grafana charts making use of the [i]rate function aren't displaying any data.

How should the problem be solved?

This happens when time duration parameters in promql queries are smaller than the scrape interval Prometheus is configured with. The solution is detailed in this blog post, which basically states that one should always use $__rate_interval instead of hard-coding those duration parameters, but this also requires properly declaring in Grafana (under config->data sources) the scrape interval that Prometheus is using. So fixing this entails updating the dashboards under grafana/dashboards/ and when those get approved, sync them with https://grafana.com/orgs/linkerd . Also the Grafana doc in the website should be updated, pointing out the scrape interval setting, noting that by default the Grafana settings in grafana/values.yaml disable admin access (disable_login_form: true), which would need to be changed (at least temporarily) in order to access the admin settings.

Any alternatives you've considered?

Nope.

How would users interact with this feature?

No response

Would you like to work on this feature?

No response

[dmitryzykov]

+1 for this issue. It should be fixed in dashboards or at least highlighted importance of having 10s scrape interval in prometheus scraping rules in documentation.

I use kube-prometheus-stack as external prometheus, which has default scrape interval 30s and grafana doesn't show data for linkerd dashboards.

After changing scrape interval to 10s linkerd dashboards in grafana show all data properly.

[TarekAS]

I've imported the dashboards to my Grafana Cloud setup and I'm facing this issue. The default scrape interval is 60s, I'm hesitating to reduce it since it could double the costs.

Any suggestions?

[adleong]

Hi @TarekAS. You'll want the duration parameter of your queries to be longer than your scrape interval. You can use whatever scrape interval is appropriate for your use-case and budget, but you may need to increase the duration parameter in your queries to be longer as a result.

[TarekAS]

Thanks for the reply @adleong . Changing the duration parameter on every query requires the modification of almost every graph in the provided linkerd dashboards. There should be a way to set it via a variable to make this a bit easier.

In any case, I just fixed it by decreasing the scrape duration.