prometheus-boshrelease
prometheus-boshrelease copied to clipboard
cloudfoundry
Grafana APP dashboard Issue: No data even after upgrade from 26.2.0 to 26.4.0
Hi Team
We face issue only on the below dashboard with NO Data, referred the below Fix CF Dashboard Issues #407 and we upgraded our environment from 26.2.0 to latest version 26.4.0
Still the below dashboard are showing no data: Apps:Latency (no data) Apps: Requests (no data) Apps: System (incomplete data) CF:Organization Memory Quotas (no data for "Organization's Memory Quota Consumption") CF:Route Emitter (no data) CF:Summary (no data for "Users") Component Metrics (no data)
In prometheus able to see below error as well:
Get "http://x.x.x.x:9193/metrics": context deadline exceeded
Kindly help on this, its affecting our deployments on day to day basis.
Hi @Murugan2all ,
what happens if you call the http://x.x.x.x:9193/metrics yourself e.g. via curl? Does it show you metrics (it maybe takes some time). If you should be able to retrieve metrics like that but it takes long until you receive a response you could try to adjust the scrape config for the cf-exporter to scrape less frequent and increase the scrape timeout for the exporter.
@benjaminguttmann-avtq : I am able to get the output for http://x.x.x.x:9193/metrics. Below is the configuration set for cf_exporter for now.
scrape_interval: 4m scrape_timeout: 2m
Data is intermediately working for now. How to fix this permanently?
@Murugan2all In our environments we apply an ops file to persist such a config, because if your environments become bigger you might need to increase that intervals from time to time.
In large environments, cf_exporter has trouble answering prometheus before its scrap timeout. This should be fixed be #445
@Murugan2all we bumped cf_exporter with v28.0.0 which brings lots of performance improvements. Please try that version out and see if that solves your issue.
Assuming that v28 fixes your problem, I'm closing this issue. Feel free to reopen if the problem persists