Grafana's /alerting/list doesn't show any alerts via datasource plugin

[chouzee]

Hi! I try to list alerts in Grafana using the plugin as a datasource but nothing shows up but they are show when I don't use the plugin. The url for both datasources are the same. Could you clarify if the plugin supports this feature?

[chouzee]

Forgot to mention: for both datasource I turned on Manage alerts via Alerting UI

[dmitryk-dk]

Hi @chouzee ! I have checked your question.

1. I created a test alert

2. Went to alerting tab

3. I Checked the List tab, and I had no error here

[dmitryk-dk]

@chouzee can you share what version you use? Your error said that something was wrong with the network.

[yuriydzobak]

@dmitryk-dk this one feature https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1739 and https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2583 like, read rules, alerts from VMalert. It worked in datasource prometheus

[dmitryk-dk]

@yuriydzobak ! Thank you, I will look at those issues and test them.

[hagen1778]

@dmitryk-dk I can confirm alerting rules aren't working with VictoriaMetrics DS plugin. It needs to be fixed. It is likely, plugin doesn't forward alerts request to the VictoriaMetrics backend.

Steps to reproduce:

1. Setup Grafana with VictoriaMetrics plugin
2. Spin up VictoriaMetrics single node ./bin/victoria-metrics --vmalert.proxyURL=http://localhost:8880
3. Spin up vmalert with some alerting rules ./bin/vmalert -rule=/alerting/rules -datasource.url=http://localhost:8428 -notifier.url=http://localhost:9093
4. Open Grafana and go to Alerting tab

Expected result:

Actual result:

Please note, both datasources are pointing to the same URL

[dmitryk-dk]

Hi @hagen1778 , @yuriydzobak , @chouzee ! I have found the difference between grafana plugin and our implementation. I think we have a possible solution and @Loori-R will try to do it on FE part

[dmitryk-dk]

Hi @hagen1778 , @yuriydzobak , @chouzee ! I have investigated the issue and created two issues in the Grafana repository. The main problem is when data source type differs from prometheus or loki grafana only calls internal API for alerts created from the Grafana UI. There are two issues with details explanations. https://github.com/grafana/grafana/issues/67466 https://github.com/grafana/grafana/discussions/68169

[hagen1778]

So it is like Grafana doesn't make the API call to fetch the alerts if your datasource isn't prometheus or loki? Is my understanding correct?

[dmitryk-dk]

So it is like Grafana doesn't make the API call to fetch the alerts if your datasource isn't prometheus or loki? Is my understanding correct?

They make API call, but they use the default source name grafana when to build URL to fetch

`/api/prometheus/${getDatasourceAPIUid(dataSourceName)}/api/v1/rules`,

function getDatasourceAPIUid(dataSourceName) returns grafana instead of the datasource_ID.

But even if we do some tricks with Grafana code the backend code has some conditions which prevent the request to data source different from Prometheus or loki. When I remove those conditions and add datasource type like victoriametrics to the places where grafana checks datasource type it works as expected.

[vainkop]

That seems like a fix for the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1739#issuecomment-1118394988

  -vmalert.proxyURL string
     Optional URL for proxying alerting API requests from Grafana. For example, if -vmalert.proxyURL is set to http://vmalert:8880 , then requests to /api/v1/rules are proxied to http://vmalert:8880/api/v1/rules

[dmitryk-dk]

vmalert.proxyURL Hi @vainkop ! I don't think this will help, because Grafana itself determines where to send the request. The problem is that Grafana stores encoded data that supports aletings

[vainkop]

vmalert.proxyURL Hi @vainkop ! I don't think this will help, because Grafana itself determines where to send the request. The problem is that Grafana stores encoded data that supports aletings

So at the moment the Grafana (6.59.*) used in https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-k8s-stack can be considered as not full compatible with VMalert (v1.94.0) from the same chart? :(

[jordangarside-kiwi]

Hi I just wanted to confirm if this was fixed or not because I'm still seeing the issue. I have vmalert.proxyURL configured for my VM Single Servers, and both the Prometheus and VM datasources are pointing at the same vmauth URL.

VM Plugin Version: 0.13.3

[hagen1778]

Hello @jordangarside-kiwi!

No, this issue can't be fixed by us - see https://github.com/VictoriaMetrics/victoriametrics-datasource/issues/59#issuecomment-1541456768. Alerting UI in Grafana works only with datasources with IDs prometheus or loki. No other datasource can have this ID, hence no other datasource can use alerting UI.

We contacted Grafana support and were told the change in this logic isn't happening.

[chocholom]

Hi @jordangarside-kiwi, as a workaround you may configure a Prometheus datasource in Grafana. For single-node Victoria it's pretty straight forward as described by @hagen1778 in https://github.com/VictoriaMetrics/victoriametrics-datasource/issues/59#issuecomment-1482542960 If you are using cluster setup you may use vmauth https://docs.victoriametrics.com/vmauth/#per-tenant-authorization with at least this minimalistic config

unauthorized_user:
  url_map:
  - src_paths:
    - "/api/v1/query"
    - "/api/v1/query_range"
    - "/api/v1/series"
    - "/api/v1/labels"
    - "/api/v1/label/.+/values"
    - "/api/v1/status/buildinfo"
    - "/select/.*"
    url_prefix:
    - "http://yourvmselectnode-1:8481/select/0/prometheus"
    - "http://yourvmselectnode-2:8481/select/0/prometheus"
  - src_paths:
    - "/api/v1/rules"
    - "/api/v1/alerts"
    url_prefix:
    - "http://yourvmalertnode-1:8880/"
    - "http://yourvmalertnode-2:8880/"

Or you may use nginx or other tool that you are using for routing the requests within your cluster accordingly. After that point the new prometheus datasource to the port of vmauth/nginx. Datasource managed rules should appear in Grafana.

Obviously this new data source won't know MetricsQL. so MetricsQL syntax may show you warnings etc :(

[jordangarside-kiwi]

@chocholom

Yeah right now I set up vmalert.proxyURL in the vmsingle servers, but I think your approach through vmauth is better because,

• vmalert.proxyURL only supports 1 vmalert URL
• you don't end up with an extra hop through vmsingle servers

Thanks!

[jordangarside-kiwi]

Does anyone know if it's possible to use the loki provider with Victoria Logs to pull in those log alerts? If so what URIs are queried?

[tiny-pangolin]

Does anyone know if it's possible to use the loki provider with Victoria Logs to pull in those log alerts? If so what URIs are queried?

Supporting the loki query language Victorialogs is not supported at this time, but there is an open issue for it. If you need to view alerting and recording rules from Victorialogs in Grafana, it is possible to use vmalert and with Victoriametrics with -vmalertProxyURL set and all vmalert rules will be displayed in the UI not matter if they are metricsql or logsql queries.

[jordangarside-kiwi]

If you need to view alerting and recording rules from Victorialogs in Grafana, it is possible to use vmalert and with Victoriametrics with -vmalertProxyURL set and all vmalert rules will be displayed in the UI not matter if they are metricsql or logsql queries.

This is the thing that wasn’t working for me. I’ll double check my setup.

[tiny-pangolin]

If it helps here are the docs for vmalert for Victorialogs. There's also an example for using 1 vmalert for metrics and logs https://docs.victoriametrics.com/victorialogs/vmalert/