Add log message in alert

[saada]

Is your feature request related to a problem? Please describe. We need to retrieve contents off of the matching alert rules. Currently alert rules take in metric LogQL queries and report either true or false. If the query is true, it notifies alertmanager and all is good.

However, if we wanted to send the contents of the logs that match the query, it would almost need to be a separate field to choose the exact logs that would be sent as part of the notification message.

Describe the solution you'd like

Solution 1: Infer the log query from the metric query

This is better because the user won't have to change any existing alert rules and can simply update the alert message render template. A new {{ $log := range .Logs }} is magically available.

Solution 2: Allow user to add their own custom log query in addition to metric query

If it's hard to derive log message from a metric query, Loki ruler could take two queries.

1. A metric query for triggering the alert itself
2. An optional log query to pass in to the message template such as {{ $log := range .LogMessages }}

Describe alternatives you've considered The only alternative was to build my own ruler? Or use some other solution like Graylog that already supports this feature.