target discovery: discovering authorization parameters

[redbaron]

Use case: scrape pprof from multiple pods in multiple namespaces with unique HTTP auth params.

Current discovery, being a prometheus discovery, doesn't seem to support "dynamic" or "discoverable" secrets. Job can find multiple targets, but all of them must have same HTTP auth param. Prometheus overcomes it with help from Prometheus Operator which configures separate jobs for each target, but maybe there can be something done without it?

It would be great if there was a way to specify a way to discover auth params relative to discovered target.

something like:

basic_auth_discovery:
    targetRef:
       apiVersion: "core/v1"
       kind:       Secret
       name:     xyz
       namespace: ${target_namespace}
    username: ".config | fromYaml | .username" // jq syntax
    password: ".config | fromYaml | .password"

that is first targetRef is used to locate kubernetes resource using some metadata from discovered target. Then username and password are extracted from that resource. Here I used jq syntax which which takes resource json as an input and produces string with the username and password. In actual implementation something available in Go like jsonnet or cue or maybe even Go template with some custom funcs can be used.

[brancz]

Here we control all parts, so I wouldn't be against having some deeper Kubernetes integrations. Sort of as if the Prometheus Operator functionality was built into Prometheus directly. I do wonder what other Prometheus maintainers would think of doing this directly in Prometheus as well though, I'm not sure how good it would be if we diverged significantly from Prometheus, having the same patterns was very intentional.

[redbaron]

One way to make it less intrusive is to have a service which does proxying to pprof endpoints into different namespaces and handles secret discovery itself. For that to work, parca need just a small change to allow building unique URL to that proxy service for each target it tries to scrape.

Something like scrapeUrl: https://pprof-proxy/${target.namespace}/${target.pod}/${target.port} , it is then up to that proxy service to proxy request to the relevant pod and handle authnetication