ruler: support explore URL format in the GeneratorURL

[FUSAKLA]

From original discussion on slack

Current state

In Prometheus, an alert does have a GeneratorURL which points to the graph view of the particular Prometheus which sent it, with the query from the alert already filled in.

With Mimir ruler this is possible the same way: Currently, ruler allows configuring the external_url and then it generates the GeneratorURL using the Prometheus code which means appending /graph?g0.expr=%s&g0.tab=0 to the external_url.

Problem

The issue is that Mimir does not have the Prometheus compatible UI, so there is nowhere to point the URL to. Naturally, I'd expect this to be compatible with the Grafana explore since that is the native UI for querying Mimir, but it appears not.

Suggestion

Add probably 2 new configuration options to the ruler something like

ruler:
  external_url: <Grafana URL>
  # Default prometheus so no breaking change, if set to grafana-explore, generator_datasource_id would be required
  generator_url_format: [prometheus(default),grafana-explore]
  # Explore requires setting the datasource ID in the URL if I'm not mistaken, so it would have to be configurable here
  generator_datasource_id: xyz

Workaround

I found quite easy running only Prometheus react UI pointing to a Mimir. This way I'm capable of doing an "adapter" to make the URL work (and as a side effect give users the classic Prometheus UI which would ease the Migration for them)

If agreed on the solution, I'd be happy to implement it :)

[pracucci]

You raised a very good point, and I agree with you that the Mimir ruler should be able to generate URLs which work with Grafana. However, I'm wondering if we should integrate it with Grafana Alerting, instead of the Grafana explore, but I don't have any concern supporting Grafana explore too (or begin just supporting Grafana explore).

Few comments about your proposal:

• generator_url_format: I'm wondering if we should call it something like generator_url_backend because it specifies the type of backend.
• generator_datasource_id: I would check if we can avoid it (e.g. relying on the default one). If not, I would rather call it generator_url_grafana_datasource_id to clarify it's the Grafana datasource it used in the generator URL.

[FUSAKLA]

Right, linking to Grafana Alerting would probably make sense too, but I don't have much experience with that yet so I'm not able to suggest how that would work.

Regarding your comments on the proposal, both sound reasonable and make the configuration clearer, so :+1:.

Regarding the datasource, if you set "datasource":"default" in the Explore URL it works, but I do recall that those datasource references are being somehow revisited, so not sure if this won't be deprecated in the future. But I do think it would be better to make this configurable for cases such as having 2 Mimir clusters in a single Grafana instance or something like that.

[FUSAKLA]

We eventually ended up running just the React part of Prometheus UI pointing to a Mimir API, and it works really well and easy to make and solves the issue for us in combination with the injection of tenant ID in the alert template.

But if we agree the way to do this, I'd be happy to implement it

[Packetslave]

We do this in a Lambda (which is fed JSON-formatted alerts from Alertmanager via SNS). Here's the relevant Python code. Not perfect, but it mostly works for us. Hope it's helpful to someone.

def prometheus_graph_url_to_grafana_explore(input):
    parsed = urlparse(input)
    qs = parse_qs(parsed.query)
    expr = qs["g0.expr"][0]

    out = {
        "datasource": "Prometheus (OBS APS instance)",
        "queries": [
            {
                "refId": "A",
                "expr": expr,
                "editorMode": "code",
                "range": True,
                "instant": False,
                "format": "time_series",
                "exemplar": False,
                "interval": "",
            }
        ],
        "range": {"from": "now-12h", "to": "now"},
    }

    return f"{GRAFANA_URL}/explore?orgId=1&left={quote(json.dumps(out))}"

[FUSAKLA]

In theory, it could be an additional template function since it is quite easy to add it after changes being made in https://github.com/grafana/mimir/pull/3758

Something like grafanaExploreURL <grafana_url> <datasource> <from> <to> <expr>

That would leave the generatorURL of the alert to be still useless, but would simplify construction of the explore URL :shrug:

[pracucci]

I like the idea of a custom function. We can still leave the door open to properly fix generatorURL but having the option to also generate the explore URL via function looks neat to me.

[FUSAKLA]

Ok, I'll create new PR with the agreed function.

Thanks @Packetslave, your example will make it easier and hopefully this will make it easier for you also.

[FUSAKLA]

Here, as promised https://github.com/grafana/mimir/pull/3849

[pracucci]

Closed by #3849.

[FUSAKLA]

Yes, closing as done :tada: