helm-charts

helm-charts copied to clipboard

Describe the bug a clear and concise description of what the bug is.

Installing the prometheus-kube-stack chart on minikube using the following basic commands:

minikube start --driver=docker --kubernetes-version=v1.26.5 \
 --bootstrapper=kubeadm \
 --extra-config=kubelet.authentication-token-webhook=true \
 --extra-config=kubelet.authorization-mode=Webhook \
 --extra-config=scheduler.bind-address=0.0.0.0 \
 --extra-config=controller-manager.bind-address=0.0.0.0
helm repo update
helm install prometheus-stack prometheus-community/kube-prometheus-stack --create-namespace -n monitoring

After the installation, all the dashboards in Grafana that should show CPU/Memory consumption by Pods and Namespace are failing with no data available.

What's your helm version?

version.BuildInfo{Version:"v3.10.2", GitCommit:"50f003e5ee8704ec937a756c646870227d7c8b58", GitTreeState:"clean", GoVersion:"go1.18.8"}

What's your kubectl version?

v1.25.4

Which chart?

prometheus-kube-stack

What's the chart version?

48.1.1

What happened?

No response

What you expected to happen?

No response

How to reproduce it?

No response

Enter the changed values of values.yaml?

No response

Enter the command that you execute and failing/misfunctioning.

helm install prometheus-stack prometheus-community/kube-prometheus-stack --create-namespace -n monitoring

Anything else we need to know?

No response

I am also getting the same issue with prometheus-kube-stack chart greater than 45.24.0.

Having the same issue here, I'm gonna try with a previous release version and will report back

having the same problem. could you please let me know if the problem resolved.

Same error here

minikube version: v1.31.2 kube-prometheus-stack-52.1.0 APP REVISION v0.68.0

Any updates?

I think I have figured it out. Or atleast part of it. The metrics that are used on grafana (and I assume lens, and every other dashboard) for cpu are generated here with this container_cpu_usage_seconds_total{job="kubelet", metrics_path="/metrics/cadvisor", image!=""} being the key metric.

I have an old k8s cluster with a prometheus instance that I still have producing metrics. On this instance, Some, but not all, of the cpu_usage_second values have an image label. On my new cluster, NONE have an image label.

Potentially, either the rules need to be updated to accept data without a image, or the kubelet job needs to be exporting the image label properly. I assume a change to the kubelet exporter was the root cause. Going to do more research to see if I can determine how to solve this issue moving forward.

This appears to be an issue with kubelet no longer returning some labels due to the removal of docker from k8s. See this comment in a similar issue

I managed to resolve the issue with my dashboard. The solution involved removing the additionalRuleLabels in the configuration. Here's the updated configuration:

ValuesInLine:
  defaultRules:
    # additionalRuleLabels:
    #   cluster: Masked

Additionally, I noticed some fixes in the prometheus-kube-stack Helm chart versions greater than 45.24.0. Hope this helps others facing a similar issue!

I managed to resolve the issue with my dashboard. The solution involved removing the additionalRuleLabels in the configuration. Here's the updated configuration:

ValuesInLine:
  defaultRules:
    # additionalRuleLabels:
    #   cluster: Masked

Additionally, I noticed some fixes in the prometheus-kube-stack Helm chart versions greater than 45.24.0. Hope this helps others facing a similar issue!

What configuration file did you update?

by default the additionalRuleLabels: {} is empty, might not be the fix for everyone i guess. At least in my case it wasn't - installing the stack with helm on k8s. But not sure yet how to fix it otherwise

Hi all,

can someone test, if the issues are gone with this:

minikube start --container-runtime=containerd

All production grade clusters are using containerd or cri-o as container runtime, while minikube uses docker-shim as default which works differently.

i found a solution for k8s deployment via helm chart. inside the kubelet the default is that some metrics get relabled - some cpu and memory metrics were in there cAdvisorMetricRelabelings: [], i changed it to an empty array and it works. maybe you guys can adapt this somehow?

On my local system, I work with

minikube start --container-runtime=containerd

which works out of the box.

Solution mentioned by @Hackmeat fixed my issue as well (thanks!) , here are my commands:

minikube start \
    --nodes 3 \
    --cpus 2 \
    --memory 2GB \
    --disk-size 10GB \
    -p multinode-demo \
    --container-runtime=containerd \
    --kubernetes-version=v1.28.3 \
    --bootstrapper=kubeadm \
    --extra-config=kubelet.authentication-token-webhook=true \
    --extra-config=kubelet.authorization-mode=Webhook \
    --extra-config=scheduler.bind-address=0.0.0.0 \
    --extra-config=controller-manager.bind-address=0.0.0.0

helm upgrade monitoring \
	--install \
	--version 58.2.2 \
	--namespace monitoring \
	--create-namespace \
	--set alertmanager.enabled=false \
	--set grafana.enabled=true \
	--set kubernetesServiceMonitors.enabled=true \
	--set nodeExporter.enabled=true \
	--set prometheusOperator.enabled=true \
	--set prometheus.enabled=true \
	--set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \
	--set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \
	--set kubelet.cAdvisorMetricRelabelings='[]' \
	prometheus-community/kube-prometheus-stack