aws-workshop-for-kubernetes
aws-workshop-for-kubernetes copied to clipboard
201 - Prometheus not loading any metrics.
I'm currently following https://github.com/aws-samples/aws-workshop-for-kubernetes/tree/master/02-path-working-with-clusters/201-cluster-monitoring
I've successfully loaded Prometheus in a browser after using the proxy command kubectl port-forward $(kubectl get po -l prometheus=prometheus -n monitoring -o jsonpath={.items[0].metadata.name}) 9090 -n monitoring
but i am not seeing any of the metrics on localhost:9090
This is all I see:
I have gone through 201 from the beginning twice, with the same results, following the cleanup shown at the end of the tutorial
Update
I just did some digging and noticed a lot of the following in the logs for the prometheus-prometheus-0 pod:
level=error ts=2018-06-11T18:11:56.227803158Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:177: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-operator\" cannot list endpoints in the namespace \"kube-system\""
level=error ts=2018-06-11T18:11:56.231432703Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:177: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-operator\" cannot list endpoints in the namespace \"monitoring\""
level=error ts=2018-06-11T18:11:56.231436883Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:178: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-operator\" cannot list services in the namespace \"kube-system\""
level=error ts=2018-06-11T18:11:56.231516245Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:177: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-operator\" cannot list endpoints in the namespace \"monitoring\""
level=error ts=2018-06-11T18:11:56.243360069Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:177: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-operator\" cannot list endpoints in the namespace \"monitoring\""
level=error ts=2018-06-11T18:11:56.243441332Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:178: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-operator\" cannot list services in the namespace \"kube-system\""
level=error ts=2018-06-11T18:11:56.243462191Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:178: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-operator\" cannot list services in the namespace \"monitoring\""
level=error ts=2018-06-11T18:11:56.24351401Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:178: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-operator\" cannot list services in the namespace \"monitoring\""
level=error ts=2018-06-11T18:11:56.24351825Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:177: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-operator\" cannot list endpoints in the namespace \"kube-system\""
level=error ts=2018-06-11T18:11:56.243564991Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:177: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-operator\" cannot list endpoints in the namespace \"default\""
level=error ts=2018-06-11T18:11:56.243573509Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:178: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-operator\" cannot list services in the namespace \"kube-system\""
level=error ts=2018-06-11T18:11:56.243612852Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:178: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-operator\" cannot list services in the namespace \"monitoring\""
level=error ts=2018-06-11T18:11:56.243625904Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:177: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-operator\" cannot list endpoints in the namespace \"kube-system\""
level=error ts=2018-06-11T18:11:56.254559683Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:178: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-operator\" cannot list services in the namespace \"monitoring\""
level=error ts=2018-06-11T18:11:56.254651698Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:178: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-operator\" cannot list services in the namespace \"default\""
level=error ts=2018-06-11T18:11:56.254773778Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:177: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-operator\" cannot list endpoints in the namespace \"kube-system\""
level=error ts=2018-06-11T18:11:56.254851617Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:177: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-operator\" cannot list endpoints in the namespace \"monitoring\""
level=error ts=2018-06-11T18:11:56.254910369Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:178: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-operator\" cannot list services in the namespace \"kube-system\""
level=error ts=2018-06-11T18:11:56.449285051Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to watch *v1.Pod: unknown (get pods)"
level=error ts=2018-06-11T18:11:56.452019179Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to watch *v1.Pod: unknown (get pods)"
level=error ts=2018-06-11T18:11:56.452942474Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to watch *v1.Pod: unknown (get pods)"
level=error ts=2018-06-11T18:11:56.453427575Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to watch *v1.Pod: unknown (get pods)"
level=error ts=2018-06-11T18:11:56.49484826Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to watch *v1.Pod: unknown (get pods)"
level=error ts=2018-06-11T18:11:56.496102528Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to watch *v1.Pod: unknown (get pods)"
level=error ts=2018-06-11T18:11:56.498910773Z caller=main.go:212 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:179: Failed to watch *v1.Pod: unknown (get pods)"
[UPDATE] I think the issue is just that prometheus-operator and prometheus are sharing the same RBAC here. While they should be different. The other Cluster Role that should be used is: https://github.com/coreos/prometheus-operator/blob/v0.14.1/Documentation/rbac.md#prometheus-rbac and should be used as a different SA here https://github.com/aws-samples/aws-workshop-for-kubernetes/blob/master/02-path-working-with-clusters/201-cluster-monitoring/templates/prometheus/prometheus.yaml#L228
I realized that as the API Servers appeared to be down (which happened because the get
of /metrics
is not listed in the prometheus-operator cluster-role).
So the underlying problem is that there is a missing cluster role.
I also hit this issue - You can just kubectl edit clusterrole prometheus-operator -n monitoring
and add the missing verbs, in your case list
for endpoints/svcs and watch
for pods I think.
This is the RBAC that got the UI up in my case.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
[...]
rules:
- apiGroups:
- extensions
resources:
- thirdpartyresources
verbs:
- '*'
- apiGroups:
- apiextensions.k8s.io
resources:
- customresourcedefinitions
verbs:
- '*'
- apiGroups:
- monitoring.coreos.com
resources:
- alertmanagers
- prometheuses
- servicemonitors
verbs:
- '*'
- apiGroups:
- apps
resources:
- statefulsets
verbs:
- '*'
- apiGroups:
- ""
resources:
- configmaps
- secrets
verbs:
- '*'
- apiGroups:
- ""
resources:
- pods
verbs:
- watch
- list
- delete
- apiGroups:
- ""
resources:
- services
- endpoints
verbs:
- get
- list
- create
- watch
- update
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- namespaces
verbs:
- list
Interestingly, these are not required according to the doc of 0.14.1
Can someone make the changes that @CharlyF mentioned? As of 7/14/18 this was still not working.
I had the same issue and had to modify the cluster role.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus-operator
namespace: monitoring
rules:
- apiGroups:
- extensions
resources:
- thirdpartyresources
verbs:
- "*"
- apiGroups:
- apiextensions.k8s.io
resources:
- customresourcedefinitions
verbs:
- "*"
- apiGroups:
- monitoring.coreos.com
resources:
- alertmanagers
- prometheuses
- servicemonitors
- prometheusrules
verbs:
- "*"
- apiGroups:
- apps
resources:
- statefulsets
verbs: ["*"]
- apiGroups: [""]
resources:
- configmaps
- secrets
verbs: ["*"]
- apiGroups: [""]
resources:
- pods
verbs: ["list", "delete", "watch"]
- apiGroups: [""]
resources:
- services
- endpoints
verbs: ["get", "create", "update", "watch", "list"]
- apiGroups: [""]
resources:
- nodes
verbs: ["list", "watch"]
- nonResourceURLs:
- /metrics
verbs: ["get"]
- apiGroups: [""]
resources:
- namespaces
verbs: ["list"]
And to be a little more precise—if you followed the directions in the guide, and you have a blank Targets page (and the prometheus container in the prometheus-prometheus-1
pod is showing errors in the log like the ones shown earlier in this thread), then you need to:
- Copy and paste the entire code block in @dannyvargas23's comment above; paste it into the
prometheus-bundle.yaml
file directly. - Run the command
kubectl apply -f templates/prometheus/prometheus-bundle.yaml
again, to apply the changes.
After a couple minutes, you should start seeing Targets 'UP' in the Prometheus UI.
I'll file a PR with this change, hopefully it can get merged soon!