Prometheus data source for Grafana not working

[galah92]

What happened?

Installed both main & v0.12.0 versions, in both of them Grafana dashboards not showing data.

How to reproduce it (as minimally and precisely as possible):

Go through the installation instructions here.

kubectl create -f manifests/setup
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
kubectl create -f manifests/

Environment

WSL2 inside Windows 11, Using K3s locally.

• Prometheus Operator version:

  Insert image tag or Git SHA here

Name:                   prometheus-operator
Namespace:              monitoring
CreationTimestamp:      Thu, 16 Mar 2023 22:00:02 -0400
Labels:                 app.kubernetes.io/component=controller
                        app.kubernetes.io/name=prometheus-operator
                        app.kubernetes.io/part-of=kube-prometheus
                        app.kubernetes.io/version=0.62.0

• Kubernetes version information:

  kubectl version

Client Version: v1.26.2
Kustomize Version: v4.5.7
Server Version: v1.25.7+k3s1

• Kubernetes cluster kind:

  insert how you created your cluster: kops, bootkube, tectonic-installer, etc.

• Manifests:

insert manifests relevant to the issue

• Prometheus Operator Logs:

Insert Prometheus Operator logs relevant to the issue here

• Prometheus Logs:

Insert Prometheus logs relevant to the issue here

Anything else we need to know?:

Running kubectl describe pod grafana-9f58f8675-ncfch --namespace monitoring yield the following:

Events:
  Type     Reason       Age                     From               Message
  ----     ------       ----                    ----               -------
  Normal   Scheduled    4m43s                   default-scheduler  Successfully assigned monitoring/grafana-9f58f8675-ncfch to dell14
  Warning  FailedMount  4m42s                   kubelet            MountVolume.SetUp failed for volume "grafana-dashboard-pod-total" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount  4m42s                   kubelet            MountVolume.SetUp failed for volume "grafana-dashboard-scheduler" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount  4m42s                   kubelet            MountVolume.SetUp failed for volume "grafana-dashboard-prometheus" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount  4m42s                   kubelet            MountVolume.SetUp failed for volume "grafana-dashboard-nodes" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount  4m42s                   kubelet            MountVolume.SetUp failed for volume "grafana-dashboard-kubelet" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount  4m42s                   kubelet            MountVolume.SetUp failed for volume "grafana-config" : failed to sync secret cache: timed out waiting for the condition
  Warning  FailedMount  4m42s                   kubelet            MountVolume.SetUp failed for volume "grafana-dashboard-k8s-resources-cluster" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount  4m42s                   kubelet            MountVolume.SetUp failed for volume "grafana-dashboard-k8s-resources-workload" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount  4m41s                   kubelet            MountVolume.SetUp failed for volume "grafana-dashboard-k8s-resources-workloads-namespace" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount  4m41s                   kubelet            MountVolume.SetUp failed for volume "grafana-dashboard-controller-manager" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount  4m41s (x2 over 4m42s)   kubelet            MountVolume.SetUp failed for volume "grafana-dashboard-workload-total" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount  4m41s                   kubelet            MountVolume.SetUp failed for volume "grafana-dashboard-namespace-by-pod" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount  4m41s                   kubelet            MountVolume.SetUp failed for volume "grafana-dashboard-proxy" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount  4m41s                   kubelet            MountVolume.SetUp failed for volume "grafana-dashboard-apiserver" : failed to sync configmap cache: timed out waiting for the condition
  Warning  FailedMount  4m38s (x10 over 4m42s)  kubelet            (combined from similar events): MountVolume.SetUp failed for volume "grafana-dashboard-prometheus" : failed to sync configmap cache: timed out waiting for the condition