Add ServiceMonitor (and PodMonitor) resources instead of JUST `prometheus.io/` annotations

[frittentheke]

trafficstars

Proposed change

Full disclose: This has been asked for >3 years ago in #2029 already. But I strongly believe this is a (still) valid idea and a great improvement to this powerful chart to run JupyterHub on K8s.

I'd like to propose adding ServiceMonitor (https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.ServiceMonitor) and potentially also PodMonitor (https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.PodMonitor) resources to integrate the exposed metrics with Kubernetes environments using the Prometheus Operator (https://github.com/prometheus-operator/prometheus-operator)

While I understand that you don't want to support all sorts of individual configurations, the usage of those Custom Resources to configure Prometheus is really really common.

If you look across the Kubernetes ecosystem e.g. the Prometheus-Community repo with charts for various exporters, but also also other applications meant to run on Kubernetes ...

• Kube-State-Metrics https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-state-metrics/templates/servicemonitor.yaml
• Grafana https://github.com/grafana/helm-charts/blob/main/charts/grafana/templates/servicemonitor.yaml
• NGINX Ingress Controller https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/templates/controller-servicemonitor.yaml
• qdrant vector DB https://github.com/qdrant/qdrant-helm/blob/main/charts/qdrant/templates/servicemonitor.yaml
• OpenSearch https://github.com/opensearch-project/helm-charts/pull/537
• Cloud Native PG PostgreSQL Operator https://cloudnative-pg.io/documentation/1.17/monitoring/
• TopoLVM CSI https://github.com/topolvm/topolvm/blob/main/charts/topolvm/templates/node/podmonitor.yaml
• RabbitMQ Operator https://www.rabbitmq.com/kubernetes/operator/operator-monitoring#prom-operator
• [...] https://github.com/search?q=repo%3Aprometheus-community%2Fhelm-charts+.Values.serviceMonitor.&type=code

The Prometheus Operator has also been adopted by lots of managed solutions, be it with the hyperscalers (Amazon EKS, Google Kubernetes Engine (GKE) Azure Kubernetes Service (AKS), ...) or on premise (VMware Tanzu Kubernetes Grid (TKG), OpenShift, Rancher)

Alternative options

The current set of annotations the likes of prometheus.io/scrape: true could remain in place and a new option in values.yaml could simply switch from those to dedicated ServiceMonitor and PodMonitor custom resources.

Who would use this feature?

Everybody using the Prometheus Operator to run their monitoring stack.

(Optional): Suggest a solution

see comment https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/2029#issuecomment-1036192230 by @dmpe including a code snippet.

I gladly push a PR, if you somewhat agree that (optionally) providing Service- and PodMonitors (or active probes even ;-) ) is a good idea.