Add namespace_selector to prometheus.operator Flow components to allow resource discovery based on namespace labels

[bcrisp4]

Request

Forgive me if this is already possible and I have missed something...

I'd like to be able to use Grafana Agent to discover all Prometheus Operator custom resources (PodMonitor, ServiceMonitor, Probe) from namespaces with a matching label.

Example:

prometheus.operator.podmonitors "pods" {
    forward_to = [prometheus.remote_write.mimir.receiver]
    namespace_selector {
        match_expression {
            key = "tenant"
            operator = "In"
            values = ["tenant-1"]
        }
    }
}

I would expect the above configuration to discover all PodMonitor resources in any namespace with the tenant=tenant-1 label.

I believe the prometheus.operator components can currently only discover resources based on labels on the resources themselves, not the containing namespace.

This sort of workflow is already possible using the Prometheus Operator. And, after a brief look at the docs, I think it is also possible in Grafana Agent when using the Operator deployment model (i.e. podMonitorNamespaceSelector is exposed here).

This functionality exists for PrometheusRule resources in the mimir.rules.kubernetes component, which may serve as an example of how it could be implemented in prometheus.operator.*.

Use case

I operate a multi-tenant Kubernetes environment. I would like to provision Grafana Agent instances for each of my tenants which automatically scrapes metrics for them and stores them in Mimir.

Tenants configure their Agent using Prometheus Operator CRs that they deploy in their own namespaces/that come bundled with 3rd party Helm charts.

In order to 1.) simplify the configuration for the tenant, and 2.) avoid one tenant accidentally polluting another tenants metrics, I would prefer each Agent to discover CRs based on namespace metadata (which I control) rather than CR metadata (which the tenant controls).

I can think of ways around this using using admission controllers / security policies to enforce labels on the CRs, but that feels a little fiddly.