Investigate Prometheus rule upstream

[aarnq]

What should be investigated. What technology/product/concept should be investigated?

With Thanos Ruler we are dependent on being able to control which Prometheus rule are evaluated where, and to ensure that they keep the cluster label in aggregations. This has led to us manually patching the rules provided by upstream/kube-prometheus-stack copied over into charts/prometheus-alerts, which will be a lot of work to maintain during upgrades.

We should investigate if we can change the upstream for these rules, possibly the upstream of kube-prometheus-stack kubernetes-mixin, and render them into the yaml we want.

What artifacts should this produce. How should the knowledge gathered from this task be captured.

A procedure to upgrade our Prometheus rules with minimal effort.

[crssnd]

take also a look at https://github.com/elastisys/compliantkubernetes-apps/issues/1941