Not able to see data with dashboard

[neel5481]

Hi,

Thanks for the dashboard. I have imported the dashboard but not able to see the data on the graph.

Here in the dashboard, I can see, we are querying "prometheus" to get those information. I can not see "alertmanager_build_info" on prometheus exposed metrics which is "http://localhost:9090/metrics".. but i can see with alertmanager exposed metrics as it is porvided by alertmanager.

I have below output from http:localhost:9093/metrics from alertmanager but not in prometheus.

alertmanager_build_info{branch="HEAD",goversion="go1.14.4",revision="4c6c03ebfe21009c546e4d1e9b92c371d67c021d",version="0.21.0"} 1

Here how are are fetching the data from prometheus ? Is something i am missing ?

@FUSAKLA Any pointer will be appreciated.

[nzrunner]

I have exactly the same issue as the OP. Does anyone have a suggestion on how to resolve this?

[FUSAKLA]

Hi, sorry for the delay :/

Just to be clear: Grafana is querying the Prometheus API not the /metrics directly. If you want to see the data you need to crape the Alertmanager's metrics with the Prometheus which is configured as a datasource for the dashboard.

Otherwise, I just found out that the dashboard JSON is maybe wrongly exported without having variable datasource.

If you import the dashboard, are you asked to choose the Prometheus datasource to be used?

[FUSAKLA]

I created PR which should hopefully fix this issue You can test the dashboard here json https://raw.githubusercontent.com/FUSAKLA/alertmanager-grafana-dashboard/0b2715d44ad0b880b0fe4b6282593d38c3ff9604/dashboard/alertmanager-dashboard.json

Please let me know if that helped

[PhillypHenning]

It did not :(

[cmonty14]

I have added another datasource from here. Review attached screenshot for details.

However I cannot select any other datasource in the Alertmanager dashboard. And this means not data is displayed.

Please advise how to fix this.

[dmitry-mightydevops]

Same issue with grafana 8.1.2

[FUSAKLA]

I'm not sure what exactly the issue is? The first report sounded more like improperly configured Prometheus.

The data source you have to use MUST be a Prometheus, not Alertmanager datasource, The Alermanager datasource is used for displaying alerts and silences in Grafana not showing metrics.

[ansonhoyt]

I had the same issue trying this dashboard on a fresh Prometheus install. After reading here, I realized Prometheus just needed to start scraping Alert Manager metrics. Now it works!

Since it wasn't being scraped and stored by Prometheus, it couldn't be queried by the dashboard.

Details

...in case this helps someone:

I noticed alertmanager_build_info isn't in Prometheus metrics (localhost:9090/metrics), but is in Alert Manager metrics (localhost:9093/metrics).

I was just scraping Prometheus and Node exporter, so added an Alert Manager scrape config:

#
# Ansible managed
#
# http://prometheus.io/docs/operating/configuration/

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s

  external_labels:
    environment: prom1.test

rule_files:
  - /etc/prometheus/rules/*.rules

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - localhost:9093

scrape_configs:
  - job_name: node
    file_sd_configs:
    - files:
      - /etc/prometheus/file_sd/node.yml
    basic_auth:
      username: admin
      password: secret
  - job_name: prometheus
    static_configs:
    - targets:
      - localhost:9090
    basic_auth:
      username: admin
      password: secret
+  - job_name: alertmanager
+    static_configs:
+    - targets:
+      - localhost:9093

A few seconds later, the dashboard starting refreshing with the Alert Manager metrics.