Allow extra relabelings to all service and pod monitors when scraped by Prometheus

[QuentinBisson]

Description

Fixes https://github.com/prometheus-operator/prometheus-operator/issues/3255

This PR allows users of the operator to ensure a set of custom relabelings to all their pod/service monitors.

This would supersed this PR https://github.com/prometheus-operator/prometheus-operator/pull/5972

Notable examples would be adding the node label using __meta_kubernetes_pod_node_name but it could be used for other organization purposes such as:

- sourceLabels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
  action: "replace"
  targetLabel: app

This feature could in the future be added to Scrape Classes as well

Type of change

What type of changes does your code introduce to the Prometheus operator? Put an x in the box that apply.

• [ ] CHANGE (fix or feature that would cause existing functionality to not work as expected)
• [x] FEATURE (non-breaking change which adds functionality)
• [ ] BUGFIX (non-breaking change which fixes an issue)
• [ ] ENHANCEMENT (non-breaking change which improves existing functionality)
• [ ] NONE (if none of the other choices apply. Example, tooling, build system, CI, docs, etc.)

Changelog entry

Please put a one-line changelog entry below. This will be copied to the changelog file during the release process.

Allow addition of extra relabelings to all service and pod monitors

[nicolastakashi]

Thanks @QuentinBisson for this PR! I personally have mixed feelings about this, because I think it should be part of scrape class as we already have proposed. I'm curious to what the rest of the team thinks about this.

[QuentinBisson]

I do think it makes sense to be able to configure this at the scrape class level but what if you want to enable it at an individual prometheus level ? You would need to configure a custom scrape class for this Prometheus ?

Also, as mentionned in my other PR, scrape classes are still at the proposal level which will take some amount of time before it lands in the operator when this would fix a real issue of ours right now and I hope others as well.

I would think the addition of scrape classes will make us transition to some kind of v2 in the Prometheus API as some fields in the Prometheus CR might be deprecated at some point so maybe that could be addressed then as well ?

[nicolastakashi]

I'm thinking of a more long-term goal, I mean adding a field to the CRD which will be deprecated very soon in favor of a feature that is under design, it might introduce a lot of complexity on the maintenance side, because we will need to introduce deprecations, breaking changes and maintain "duplicated" features.

But I do understand your POV and I'm not totally against this approach, it might be a solution indeed, just wondering if looking for a long-term makes sense since the Scrape Class proposal is almost proposed.

I think it makes sense to configure this at the scrape class level, but what if you want to enable it at an individual prometheus level? Would you need to configure a custom scrape class for this Prometheus?

Yes, from my understanding the Scrape Classe feature will be placed on the Prometheus CR level so that you can have different configs for different Prometheus.

[QuentinBisson]

Hello @nicolastakashi I fully understand your point about operator maintenance and I did not know it was coming so soon. Thinking about it over the weekend, I still think the Prometheus CR should have this property as I would envision a lot of organisations setting relabelings at the prometheus level for organizationnal stuff and maybe add custom relabelings at the scrape class level. Also, having a global field at the prometheus level would avoid the need to set the same relabelings across all scrape classes.

For instance, organizations could add the following default at the prometheus level:

• organization label
• app label
• cluster label
• other organizational labels

At the scrape class level, depending on your use case you could:

• Have a scrape class per team with a different relabeling for the team label so teams would be able to just use the scrape class and not care about setting the team label
• There are probably a lot of other labels that would make sense here but the team one was the only one I could think of 😓

But as you said, maybe let's check if other people have other ideas :)

BTW, i'd be happy to add the code for the relabelings on the scrape class level as well once it's out

[QuentinBisson]

@ArthurSens and @paulfantom as you both were on my previous PR, would you mind taking a look at this PR when you have some free time ? I would love your extra input :)

[peter-crist]

I'm extremely interested in this as well. I have a single prometheus instance and alerting solely through Grafana.

If you're not setup as a federated/remote-storage configuration, or alerting from Alertmanager, external labels don't help you and it is quite difficult to get cluster information available in Grafana. The ability to add a specific label to every metric would solve this problem. Setting up this relabel config on service/podmonitors individually is not maintainable.

Additionally, and outside the scope of this issue, Grafana doesn't provide a way of provisioning a global variable that can be used on all dashboards. This means, the solution would have to come from applying the label to the metric itself as opposed to decorating it in Grafana.

This would go a long way in the simple challenge of displaying cluster information alongside our metrics.

[QuentinBisson]

Closing this PR in favor of https://github.com/prometheus-operator/prometheus-operator/pull/6379 in scrape classes