Feature request: Support Basic Authentication in PodMonitoring

[zoeponta]

I want to use basicAuth in metrics endpoints of PodMonitoring, but GMP does not support it .

It is already implemented in GMP's AlertmanagerEndpoint and the Prometheus Operator's ServiceMonitor. So, could you please support same feature in PodMonitoring.

Thanks

[pintohutch]

Hi @zoeponta,

We agree and are working on authentication support in PodMonitoring. We can update here when this is addressed.

Thanks for your interest.

[akettmann-e24]

Just chiming in that I also would love to have the ability to even just specify headers that the request passes along, that would get me by for now so I can satisfy basic auth.

[Vaitheesh-SS]

Hi,

Does this enhancement is in place for Basic Auth on prometheus metrics endpoints? It is much needed now, since made GCP made managed prometheus will be on by default in new GKE standard clusters.

[lyanco]

There are no scrape configurations installed by default in new clusters, even with managed Prometheus on by default. Therefore as long as you don't install any scrape configs you won't violate any TLS requirements in your cluster.

[marandalucas]

We are affected by this.

We can not scrape Prometheus metrics in GCP using Managed Prometheus in our GKE cluster.

We are trying to scrape metrics from MongoDB following the following example: https://github.com/mongodb/mongodb-kubernetes-operator/blob/93c86675b851e7d44d5bfbd65375602dd4b3ad49/docs/prometheus/mongodb-prometheus-sample.yaml#L59-L104

In this scenario, we can not use "authentication", because unfortunately is not allowed in "PodMonitoring or ClusterPodMonitoring" -> https://github.com/GoogleCloudPlatform/prometheus-engine/blob/v0.5.0/doc/api.md#scrapeendpoint

[pintohutch]

Hi @marandalucas,

Thanks for raising and expressing your interest in this feature. You are correct that we indeed do not support Basic Auth...yet.

We are still scoping out this feature and are very much interested in getting it into your hands! For the time being, you can still leverage GMP in a "self-deployed" model to use the ServiceMonitor verbatim in prometheus-operator. Please see our docs for how to set that up.

Hope this helps.

[yqlu]

Another suggestion for @marandalucas

Instead of following the mongodb-kubernetes-operator example you linked, can you try setting up a standalone mongodb-exporter that uses basic auth to scrape metrics from MongoDB and exposes them in prometheus format? Then the PodMonitoring can scrape from the mongodb-exporter without any credentials, without needing to talk to MongoDB itself.

https://github.com/GoogleCloudPlatform/monitoring-dashboard-samples/pull/441/files#diff-9621388a64640b7a68fd162e21bfaf89793aee511eaca4f4d16ed98a4abfda1cR12

(These MongoDB installation exporter instructions are not yet official, but we are preparing them for publication. Let us know if they work for you!)

[marandalucas]

@yqlu WOW! You read my mind! Yes, I thought to try with "mongodb-exporter" until the new feature is released.

[chambodn]

Hi @pintohutch ,

Any updates on this feature? We use GMP and PodMonitoring resources on all our GKE clusters, but this feature prevents us from scraping metrics for one of our endpoints. I could use the self-deployed model and ServiceMonitor, as you suggested, but I would prefer to stick with the "by default" deployment.

Thanks!

[TheSpiritXIII]

Hi @chambodn this is on our roadmap for Q4 and it's probably the biggest thing we're going to prioritize! I'll start putting up PRs as soon as we finalize the design.

Thanks for the interest 🙂

[alencarsouza]

I am looking forward to this feature.

[AgustinRamiroDiaz]

Hi! I'm also looking forward for this feature

I wanted to know if there was a workaround in the meantime, like another way to configure auth? I'm trying to use ServiceMonitor but I'm not able to make it work

Workarounds I've seen proposed that don't fit my use case:

• self-deployed model of Prometheus
• creating a proxy exporter

[TheSpiritXIII]

Hi @AgustinRamiroDiaz can you let us know why self-deployment or ServiceMonitor don't work? Maybe we can help you out there.

That said, we're hard at work on this feature! We're currently proposing for Prometheus to add support for secret providers (including reading Kubernetes secrets natively) so there are a lot of moving parts here before we can get this ready. Until then, stay tuned and thanks for the support!

[AgustinRamiroDiaz]

@TheSpiritXIII thank you very much for your concern!

I ended up going with the self-deployed approach, with an almost exact replica of https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.7.4/examples/prometheus.yaml. This is great in the meantime, and I'd love to switch to full managed Prometheus when it's available :D

PS: thanks a lot for your contributions, I've been seeing all the PRs that are actively working on this :smile:

[nhat-tong]

Hi! I'm also looking forward for this feature.

We had this issue when trying to receive metrics from authorized endpoints of Nexus & Vault with GMP and PodMonitoring.

@pintohutch, @AgustinRamiroDiaz: I had setup the self-deployed collection as mentioned above (https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.7.4/examples/prometheus.yaml.)

The prometheus pod is started correctly but it seems that it cannot write metrics to GCP Cloud Monitoring due to missing permission. Here are the received errors: ts=2024-01-09T10:23:52.831Z caller=export.go:940 level=error component=gcm_exporter msg="send batch" size=200 err="rpc error: code = PermissionDenied desc = Permission monitoring.timeSeries.create denied (or the resource may not exist)." ts=2024-01-09T10:23:52.831Z caller=export.go:940 level=error component=gcm_exporter msg="send batch" size=200 err="rpc error: code = PermissionDenied desc = Permission monitoring.timeSeries.create denied (or the resource may not exist)." ts=2024-01-09T10:23:52.832Z caller=export.go:940 level=error component=gcm_exporter msg="send batch" size=200 err="rpc error: code = PermissionDenied desc = Permission monitoring.timeSeries.create denied (or the resource may not exist)."

If I understand well, the prometheus pod use the gke node's service account to write metrics into GCP Cloud Monitoring like PodMonitoring (https://cloud.google.com/stackdriver/docs/managed-prometheus/setup-unmanaged?hl=en#gmp-default-sa)? We don't know why it worked with PodMonitoring but not for this prometheus pod.

Thanks in advance for your feedback, Nhat

[AgustinRamiroDiaz]

@nhat-tong I think your problem relies on your service account configuration

If I understand well, the prometheus pod use the gke node's service account to write metrics into GCP Cloud Monitoring like PodMonitoring

I don't think so, I think it uses the service account configured in its deployment (or the default one if none is configured explicitly)

In the docs it explains how to set it up. Basically you need a Google Service Account with the Role roles/monitoring.metricWriter, and then assign that Google Service Account to the Prometheus deployment. In my case, I've assigned through Workload Identity as explained here

[nhat-tong]

@AgustinRamiroDiaz: thanks for your help. It worked well with Workload Identity.

[nhat-tong]

@AgustinRamiroDiaz: do you have any problem with Grafana when using PodMonitoring and Prometheus pod at the same time?

I'm able to scrape metrics from authenticated endpoints using Prometheus pod. However, our Grafana dashboards with PodMonitoring display N/A or NoData from time to time when this pod is started. When I stop prometheus pod, the Grafana dashboard display well as usual.

Thanks,

[AgustinRamiroDiaz]

@nhat-tong I didn't have that problem

@nhat-tong when using the self-deployed Prometheus, I'm not using PodMonitoring (since it doesn't have the auth endpoint feature). I'm using the prometheus configuration from here, but modifying the configmap to suite my needs of scraping, and also some things for RBAC (not necessary unless you want finer grained permissions)

I think your problem might be coming from trying to use both the self-deployed Prometheus and the managed one. Have you disabled the managed Prometheus? I think that self-deployed Prometheus doesn't handle the PodMonitoring resource

Another important detail is how are you configuring Grafana. Which endpoint have you connected it to?

[nhat-tong]

@AgustinRamiroDiaz: thanks for your reply. I deployed a PrometheusFrontend (https://cloud.google.com/stackdriver/docs/managed-prometheus/query-api-ui#ui-prometheus) which is like a bridge between Grafana and GCP Cloud Monitoring. So, my Grafana connected directly to this frontend.

[krishnaindani]

Is there any lead in supporting tls config for pod monitoring as it is supported in prometheus operator here?

[TheSpiritXIII]

Hey @krishnaindani we've just merged support for both BasicAuth and TLS into the main branch 🎉!

In anticipation of Cloud Next, we're pausing non-emergency rollouts. We will begin rolling out this feature to the GKE rapid release channel after the conference is over.

If you'd like, You can start using the new experimental feature by building the operator from source with DOCKER_PUSH=1 make operator on this repository to push the new docker image directly into your image registry. This change is compatible with Prometheus tag v2.45.3-gmp.2-gke.0.

[TheSpiritXIII]

Closing. This feature is now available on the GKE rapid channel on the latest GKE minor version. We'll make a proper 0.12 release on GitHub around the time we begin backporting this change to older GKE minor versions. Thanks, all!