Prometheus doesn't have permission to scrape OTel Collector in OpenShift

[fmhwong]

Bug Report

Which version of the demo you are using? opentelemetry-helm-charts b969a4f

Symptom

A clear and concise description of what the bug is.

What is the expected behavior? Following the install instruction from https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-demo#readme. Grafana OpenTelemetry Collector dashboard should show metrics.

What is the actual behavior?

Grafana OpenTelemetry Collector dashboard has no data. Examine Prometheus and found nothing under Status->Service Discovery Found this error message in Prometheus pod log

ts=2024-04-03T16:08:42.653Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:opentelemetry-demo:opentelemetry-demo\" cannot list resource \"pods\" in API group \"\" in the namespace \"opentelemetry-demo\""

Reproduce

1. helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
2. oc new-project opentelemetry-demo
3. oc create sa opentelemetry-demo
4. oc adm policy add-scc-to-user anyuid -z opentelemetry-demo
5. helm install otel-demo open-telemetry/opentelemetry-demo \
    --namespace opentelemetry-demo \
    --set serviceAccount.create=false \
    --set serviceAccount.name=opentelemetry-demo \
    --set prometheus.rbac.create=false \
    --set prometheus.serviceAccounts.server.create=false \
    --set prometheus.serviceAccounts.server.name=opentelemetry-demo \
    --set grafana.rbac.create=false \
    --set grafana.serviceAccount.create=false \
    --set grafana.serviceAccount.name=opentelemetry-demo

This issue is resolved after running oc policy add-role-to-user view -z opentelemetry-demo

We will close this issue if:

• The steps you provided are complex.
• If we can not reproduce the behavior you're reporting.

Additional Context

Environment: OpenShift version 4.15.3 with 3 master nodes and 3 worker nodes