Linkerd Prometheus: Unable to update list of all metrics: unable to fetch metrics for query unknown response code 403

[utk231]

What happened?:

Prometheus adapter not able to get metrics from Linkerd Prometheus

What did you expect to happen?:

I expected the prometheus-adapter to fetch the metrics from custom Linkerd Prometheus in the Kubernetes cluster.

Please provide the prometheus-adapter config:

prometheus: url: http://prometheus.linkerd-viz.svc
port: 9090 path: ""

rules: default: false custom: - seriesQuery: 'request_total{namespace!="",pod!=""}' resources: template: '<<.Resource>>' name: matches: 'request_total' as: "request_total" metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>, direction="inbound", deployment!="", namespace!=""}[5m])) by ( <<.GroupBy>>)'

Prometheus-adapter logs

I0725 11:44:17.036335 1 api.go:90] GET http://prometheus.linkerd-viz.svc:9090/api/v1/series?match%5B%5D=request_total%7Bnamespace%21%3D%22%22%2Cpod%21%3D%22%22%7D&start=1690285397.035 403 Forbidden E0725 11:44:17.036384 1 provider.go:229] unable to update list of all metrics: unable to fetch metrics for query "request_total{namespace!=\"\",pod!=\"\"}": bad_response: unknown response code 403

Anything else we need to know?:

Verified the correctness of the prometheus-adapter configuration. Inspected the prometheus-adapter logs with -v=6 around the time the issue occurred. Validated all the manifests and resources related to prometheus-adapter deployment. Verified the version of prometheus-adapter and prometheus. Prometheus adapter deployed without any changes to Service account, Cluster role or Cluster role binding

Environment: https://artifacthub.io/packages/helm/prometheus-community/prometheus-adapter prometheus-adapter version: 0.10.0 Cloud provider or hardware configuration: AWS EKS 1.25

[dashpole]

/assign @dgrisonnet /triage accepted

[dgrisonnet]

I0725 11:44:17.036335 1 api.go:90] GET http://prometheus.linkerd-viz.svc:9090/api/v1/series?match%5B%5D=request_total%7Bnamespace%21%3D%22%22%2Cpod%21%3D%22%22%7D&start=1690285397.035 403 Forbidden E0725

This means that prometheus-adapter can't reach your Prometheus instance because it is unauthorized. You most likely need to configure authentication on prometheus-adapter. There are a couple of options that you can use to do that: https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/cmd/adapter/adapter.go#L136-L147

[rpriyanshu9]

+1, I'm also getting the same error when trying to connect to Amazon Managed Service for Prometheus for querying metrics. I am currently using a sigv4-proxy sidecar container for signing the request and a serviceAccount for assuming a role for querying metrics from the AMP Workspace.

@dgrisonnet could you explain what type of auth do we need to configure? Do we need to provide auth details to connect to AWS AMP Workspace or something else. In my limited knowledge, I thought that having an IAM Role that allows querying operations on AWS AMP through a policy is enough for this setup (using serviceAccount annotations).

unable to update list of all metrics: unable to fetch metrics for query "{__name__=\"container:memory_usage_bytes:sum\",container!=\"POD\",kubernetes_namespace!=\"\",pod!=\"\",cluster_name=\"compute\"}": bad_response: unknown response code 403

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted