Mixin: Add experimental Loki recording rules for query analysis.

[stevesg]

Adds some experimental Loki recording rules (and some associated dashboards) to extract useful insights about costly queries from query-frontend log lines. The motivation to use Loki recording rules is as follows:

• If we added similar metrics on a per-user basis, the metrics would potentially be very high cardinality, especially if we wanted to add histograms (for computing p99, etc).
• Querying the information from Loki over large time windows becomes prohibitively slow, particularly some interesting types of analysis, such as (tenant wall time) / (total wall time) or topk() over week.

I'm sure there is plenty to discuss here, including the necessity of the rules, but I would like to propose it anyway and perhaps we can merge it in some form, to get some feedback about how useful it is from users.

Screenshots: