External Alertmanager source URLs can't link to Grafana explore

[iainlane]

When ruler.alertmanager.url is is set to point to an external Alertmanager, the source URLs sent are always in the format suitable for a Prometheus web UI, not a Grafana explore.

To Reproduce

Steps to reproduce the behavior:

1. Stand up an Alertmanager
2. Configure Mimir (I'm using GEM actually but this is the same on v2.8.0) with ruler.alertmanager.url pointing to that AM. Configure ruler.external.url to the base URL of your Grafana, and add Mimir as a datasource there.
3. Add some alerting rules which are triggering and sending over there
4. Open /alerting/list in the Grafana and expand the firing alert
5. Take a look at what "See graph" links to, it will be to a Prometheus format URL

Expected behavior

Mimir should be able to generate Grafana Explore URLs. Maybe it should even do this by default when sending externally?

Some related work was landed in #3849, maybe this could be used somehow. I think that made generating correct source URLs possible if you are using the internal Alertmanager, this is about feature parity for external AMs.

Additional Context

I believe the relevant call stack is this one

• https://github.com/grafana/mimir/blob/ddbb42c5d96c07d017a40d784c3aaed2a7caa158/pkg/ruler/compat.go#L296
• https://github.com/grafana/mimir/blob/ddbb42c5d96c07d017a40d784c3aaed2a7caa158/vendor/github.com/prometheus/prometheus/rules/manager.go#L1304-L1328
• https://github.com/grafana/mimir/blob/ddbb42c5d96c07d017a40d784c3aaed2a7caa158/vendor/github.com/prometheus/prometheus/util/strutil/strconv.go#L28-L31

Note the last two are vendored from Prometheus and the generator is hardcoded. I could see upstreaming the work from #3849 being one solution here (but license incompatibility, the author would need to consent) and changing Prometheus itself to support this. Alternatively, un-vendoring sendAlerts() and doing it all on the Mimir side.

[Skaronator]

I'm a bit surprised that a Grafana product doesn't support another Grafana product. There is a hacky workaround using templates and reReplaceAll within Alertmanager to replace the url, but it still is not ideal. Furthermore, the timestamp is missing.

Ref: https://github.com/grafana/grafana/discussions/42316 / https://github.com/grafana/grafana/issues/82943

[Rohlik]

I haven't tested it yet, but what about adding a link to Grafana Explore as part of notification templating within Alertmanager 🤔? It should be possible via grafanaExploreURL function, see https://grafana.com/docs/mimir/latest/references/architecture/components/alertmanager/#templating.

[Skaronator]

@Rohlik, this configuration relates to the Mimir Alertmanager, as referenced earlier by @iainlane in #3849. However, since we use Prometheus Alertmanager instead of Mimir Alertmanager, this config doesn't apply to us. As mentioned, changes need to be made in the Mimir Ruler code logic.

[oisin-obsec]

Has there any update on this? Grafana is very widely used and is also a Grafana product.

[holograph]

I haven't tested it yet, but what about adding a link to Grafana Explore as part of notification templating within Alertmanager 🤔? It should be possible via grafanaExploreURL function, see https://grafana.com/docs/mimir/latest/references/architecture/components/alertmanager/#templating.

The links generated by this don't seem to work (the expression is ignored in the resulting Grafana page), I'm still looking into this but it does seem like a ridiculous oversight for two products from the same stack 🤷

[ggabijaa]

Has anyone found any better workarounds for this issue?

[ggabijaa]

Created alertmanager template that is working for me so far.

Replace <grafana_endpoint>, <datasource_uid> and <org_id> with your values. Different endpoints or datasources could be dynamically selected with additional conditions based on labels.

Template does the following:

• Extracts query from GeneratorURL.
• Replaces \ with \\ and " with \" so Grafana could open the link correctly.
• Concatenates Grafana url parts with adjusted query.

{{ define "grafanaExploreURL" -}}
  {{- $grafanaURLStart := "https://<grafana_endpoint>/explore?schemaVersion=1&panes=%7B%22mx6%22:%7B%22datasource%22:%22<datasource_uid>%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22" -}}
  {{- $grafanaURLTail := "%22,%22range%22:true,%22instant%22:true,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22<datasource_uid>%22%7D,%22editorMode%22:%22code%22,%22legendFormat%22:%22__auto%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D%7D&orgId=<org_id>" -}}
  {{- if (index .Alerts 0).GeneratorURL -}}
    {{- $raw := (index .Alerts 0).GeneratorURL -}}

    {{- $strip1 := reReplaceAll ".*g0.expr=" "" $raw -}}
    {{- $strip2 := reReplaceAll "&.*" "" $strip1 -}}

    {{- $expr1 := reReplaceAll "%5C" "%5C%5C" $strip2 -}}
    {{- $expr2 := reReplaceAll "%22" "%5C%22" $expr1 -}}

    {{- printf "%s%s%s" $grafanaURLStart $expr2 $grafanaURLTail }}
  {{- else -}}
        #
  {{- end -}}
{{- end }}