Support multiple prometheus instances

[ebuildy]

What is missing?

In a case with multiple prometheus instances.

Allow in a same namespace, differents PodMonitor / ServiceMonitor to select prometheus instance, via a selector (namespaceSelector etc...).

Why do we need it?

We run a global prometheus and prometheus by team in order to separate software and business metrics.

Example: "rabbitmq must be scraped by global prometheus, whereas the nodejs app by team prometheus".

Environment

on-premise k8s cluster.

• Prometheus Operator version:

  v0.50.0

Anything else we need to know?:

[fpetkovski]

Can you use the podMonitorSelector and serviceMonitorSelector fields on the Prometheus object instead?

[ebuildy]

I am not sure, this is for prometheus instance. I would like select the prometheus namespace that will scrap the PodMonitor

[simonpasquier]

This works the other way around: pod monitors can't select which Prometheus instance(s) will scrape the targeted pods.

[ebuildy]

thanks you. Thats why I post this as a feature request.

[simonpasquier]

Not sure why the Prometheus object selecting service/pod monitors by labels doesn't fit your case. Would you mind providing more details?

[ruiengana]

Think of a setup like https://istio.io/latest/docs/ops/best-practices/observability, where Production Prometheus instance is launched using Prometheus Operator and Istio Prometheus instance is not launched using Prometheus Operator.

Does this mean we can't use Prometheus Operator to add a PrometheusRule to Istio Prometheus instance?

[simonpasquier]

@ruiengana I'm not following here: if the operator isn't managing Istio Prometheus, not sure how/why this Prometheus instance would be configured with PrometheusRule objects.

[ruiengana]

@ruiengana I'm not following here: if the operator isn't managing Istio Prometheus, not sure how/why this Prometheus instance would be configured with PrometheusRule objects.

It's clear :) I now understand that it's Prometheus CRD object with it's match labels that will look our for ServiceMonitor and PrometheusRule objects to execute.

As the creator of this issue, I was thinking that it worked in the reverse direction, e.g. to be the PrometheusRule that selected which Prometheus instance to apply.

[Punkoivan]

btw, it would be nice feature - I'm trying to build federation and looks like it's impossible with operator.

[xiu]

Hi @Punkoivan! For federation, here's an issue that'll help enable it: https://github.com/prometheus-operator/prometheus-operator/issues/5592

[simonpasquier]

@Punkoivan I'm not sure how your remark relates to this issue but it would help if you provide more details. With a few quirks you can create a federation target from a ServiceMonitor/PodMonitor.