helm-charts
helm-charts copied to clipboard
Published
20 hours ago •
prometheus-community
prometheus-community
[kube-prometheus-stack] prometheus-stack-kube-prom-operator Error : msg="http: TLS handshake error..."
Describe the bug a clear and concise description of what the bug is.
msg="http: TLS handshake error from 10.0.226.54:36916: remote error: tls: bad certificate"
What's your helm version?
3.14.2
What's your kubectl version?
1.29.2
Which chart?
kube-prometheus-stack
What's the chart version?
latest
What happened?
I'm Deploying kube-prometheus-stack with ArgoCD Application using Amazon EKS
However, after deploying kube-prometheus-stack, the following error occurred in the prometheus-stack-kube-prom-operator pod.
level=info ts=2024-03-27T05:55:07.975606693Z caller=main.go:181 msg="Starting Prometheus Operator" version="(version=0.71.2, branch=refs/tags/v0.71.2, revision=af2014407bdc25c4fc2b26cd99c9655235ebdf88)"
level=info ts=2024-03-27T05:55:07.975666128Z caller=main.go:182 build_context="(go=go1.21.6, platform=linux/amd64, user=Action-Run-ID-7656327832, date=20240125-14:57:39, tags=unknown)"
level=info ts=2024-03-27T05:55:07.97568378Z caller=main.go:193 msg="namespaces filtering configuration " config="{allow_list=\"\",deny_list=\"\",prometheus_allow_list=\"\",alertmanager_allow_list=\"\",alertmanagerconfig_allow_list=\"\",thanosruler_allow_list=\"\"}"
level=info ts=2024-03-27T05:55:07.987141023Z caller=main.go:222 msg="connection established" cluster-version=v1.28.7-eks-b9c9ed7
level=info ts=2024-03-27T05:55:08.04542884Z caller=operator.go:333 component=prometheus-controller msg="Kubernetes API capabilities" endpointslices=true
level=info ts=2024-03-27T05:55:08.070368535Z caller=operator.go:319 component=prometheusagent-controller msg="Kubernetes API capabilities" endpointslices=true
level=warn ts=2024-03-27T05:55:08.091889063Z caller=server.go:160 msg="server TLS client verification disabled" client_ca_file=/etc/tls/private/tls-ca.crt err="stat /etc/tls/private/tls-ca.crt: no such file or directory"
level=info ts=2024-03-27T05:55:08.100261947Z caller=server.go:300 msg="starting secure server" address=[::]:10250 http2=false
level=info ts=2024-03-27T05:55:08.202585326Z caller=operator.go:390 component=prometheus-controller msg="successfully synced all caches"
level=info ts=2024-03-27T05:55:08.20309889Z caller=operator.go:311 component=alertmanager-controller msg="successfully synced all caches"
level=info ts=2024-03-27T05:55:08.202599338Z caller=operator.go:428 component=prometheusagent-controller msg="successfully synced all caches"
level=info ts=2024-03-27T05:55:08.202772554Z caller=operator.go:280 component=thanos-controller msg="successfully synced all caches"
level=info ts=2024-03-27T05:55:08.215416391Z caller=operator.go:987 component=prometheus-controller key=monitoring/prometheus-stack-kube-prom-prometheus msg="sync prometheus"
level=warn ts=2024-03-27T05:55:08.63480492Z caller=klog.go:106 component=k8s_client_runtime func=Warning msg="spec.template.spec.containers[1].ports[0]: duplicate port definition with spec.template.spec.initContainers[0].ports[0]"
level=info ts=2024-03-27T05:55:08.646201453Z caller=operator.go:987 component=prometheus-controller key=monitoring/prometheus-stack-kube-prom-prometheus msg="sync prometheus"
level=info ts=2024-03-27T05:55:09.183415232Z caller=operator.go:987 component=prometheus-controller key=monitoring/prometheus-stack-kube-prom-prometheus msg="sync prometheus"
ts=2024-03-27T05:56:07.223479031Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.226.54:36902: remote error: tls: bad certificate"
ts=2024-03-27T05:56:07.23267305Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.226.54:36916: remote error: tls: bad certificate"
ts=2024-03-27T05:56:07.262879691Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.226.54:36930: remote error: tls: bad certificate"
ts=2024-03-27T05:56:07.267461471Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from
What you expected to happen?
When I connect to the domain, I am connected to the prometheus server and connected via 302 redirect to the /graph path. However, a 404 error occurs on the redirected page.
How to reproduce it?
The IP (10.0.226.54) is an IP that does not exist in the EKS resource. In my opinion, it looks like the IP came in as NAT from an external source (client).
Also, it says that the tls-ca.crt file does not exist. Is there a way to create it or solve this problem?
Currently, one ALB is set up as Ingress on the Grafana and Prometheus servers.
Below is part of the values.yaml file for the prometheus-stack chart.
Enter the changed values of values.yaml?
grafana
...
ingress:
enabled: true
ingressClassName: alb
annotations: {
alb.ingress.kubernetes.io/scheme: internet-facing,
alb.ingress.kubernetes.io/target-type: ip,
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]',
alb.ingress.kubernetes.io/certificate-arn: <AWS_CA>,
alb.ingress.kubernetes.io/success-codes: 200-399,
alb.ingress.kubernetes.io/load-balancer-name: monitoring-alb,
alb.ingress.kubernetes.io/ssl-redirect: '443',
alb.ingress.kubernetes.io/group.name: "monitoring"
}
labels: {}
hosts:
- grafana.help.store
## Path for grafana ingress
path: /
## TLS configuration for grafana Ingress
## Secret must be manually created in the namespace
##
tls: []
# - secretName: grafana-general-tls
# hosts:
# - grafana.example.com
# # To make Grafana persistent (Using Statefulset)
# #
persistence:
enabled: true
# type: sts
storageClassName: "gp2"
accessModes:
- ReadWriteOnce
size: 20Gi
------------------------------------------------------------------------------
prometheus
ingress:
enabled: true
ingressClassName: alb
annotations: {
alb.ingress.kubernetes.io/scheme: internet-facing,
alb.ingress.kubernetes.io/target-type: ip,
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]',
alb.ingress.kubernetes.io/certificate-arn: <AWS_CA>,
alb.ingress.kubernetes.io/success-codes: 200-399,
alb.ingress.kubernetes.io/load-balancer-name: monitoring-alb,
alb.ingress.kubernetes.io/group.name: "monitoring",
alb.ingress.kubernetes.io/ssl-redirect: '443'
}
labels: {}
## Redirect ingress to an additional defined port on the service
# servicePort: 8081
hosts:
- prometheus.help.store
## Paths to use for ingress rules - one path should match the prometheusSpec.routePrefix
##
paths:
- /
## For Kubernetes >= 1.18 you should specify the pathType (determines how Ingress paths should be matched)
## See https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#better-path-matching-with-path-types
# pathType: ImplementationSpecific
## TLS configuration for Prometheus Ingress
## Secret must be manually created in the namespace
##
tls: []
# - secretName: prometheus-general-tls
# hosts:
# - prometheus.example.com
Enter the command that you execute and failing/misfunctioning.
I'm Deploying kube-prometheus-stack with ArgoCD using Amazon EKS
Anything else we need to know?
There is my EKS Resources
$ kubectl get all -n monitoring
NAME READY STATUS RESTARTS AGE
pod/domain-exporter-64bf7f9949-w9b62 1/1 Running 0 2d19h
pod/prometheus-prometheus-stack-kube-prom-prometheus-0 2/2 Running 0 19h
pod/prometheus-stack-grafana-6c4bf874c5-f4vqp 3/3 Running 0 19h
pod/prometheus-stack-kube-prom-operator-776c4b9f76-9cfpp 1/1 Running 0 19h
pod/prometheus-stack-kube-state-metrics-6d555c6cb9-pd5rc 1/1 Running 0 19h
pod/prometheus-stack-prometheus-node-exporter-jps67 1/1 Running 0 19h
pod/prometheus-stack-prometheus-node-exporter-xjp7s 1/1 Running 0 19h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/domain-exporter ClusterIP 172.20.157.164 <none> 9203/TCP 2d19h
service/prometheus-operated ClusterIP None <none> 9090/TCP 19h
service/prometheus-stack-grafana ClusterIP 172.20.75.61 <none> 80/TCP 19h
service/prometheus-stack-kube-prom-operator ClusterIP 172.20.1.2 <none> 443/TCP 19h
service/prometheus-stack-kube-prom-prometheus ClusterIP 172.20.245.196 <none> 9090/TCP,8080/TCP 19h
service/prometheus-stack-kube-state-metrics ClusterIP 172.20.171.111 <none> 8080/TCP 19h
service/prometheus-stack-prometheus-node-exporter ClusterIP 172.20.217.150 <none> 9100/TCP 19h
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/prometheus-stack-prometheus-node-exporter 2 2 2 2 2 kubernetes.io/os=linux 19h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/domain-exporter 1/1 1 1 2d19h
deployment.apps/prometheus-stack-grafana 1/1 1 1 19h
deployment.apps/prometheus-stack-kube-prom-operator 1/1 1 1 19h
deployment.apps/prometheus-stack-kube-state-metrics 1/1 1 1 19h
NAME DESIRED CURRENT READY AGE
replicaset.apps/domain-exporter-64bf7f9949 1 1 1 2d19h
replicaset.apps/prometheus-stack-grafana-6c4bf874c5 1 1 1 19h
replicaset.apps/prometheus-stack-kube-prom-operator-776c4b9f76 1 1 1 19h
replicaset.apps/prometheus-stack-kube-state-metrics-6d555c6cb9 1 1 1 19h
NAME READY AGE
statefulset.apps/prometheus-prometheus-stack-kube-prom-prometheus 1/1 19h
---------------------------------------------------------------------------------
$ kubectl get all -n kube-system
NAME READY STATUS RESTARTS AGE
pod/aws-load-balancer-controller-79574fb866-45g8s 1/1 Running 0 2d21h
pod/aws-load-balancer-controller-79574fb866-f2ctk 1/1 Running 0 2d21h
pod/aws-node-dkg7t 2/2 Running 0 2d22h
pod/aws-node-vq7hx 2/2 Running 0 2d22h
pod/coredns-56dfff779f-dbvbd 1/1 Running 0 2d22h
pod/coredns-56dfff779f-hw7tv 1/1 Running 0 2d22h
pod/ebs-csi-controller-66ccf4755d-vzl5p 6/6 Running 0 2d21h
pod/ebs-csi-controller-66ccf4755d-znphb 6/6 Running 0 2d21h
pod/ebs-csi-node-khnzv 3/3 Running 0 2d21h
pod/ebs-csi-node-pw5bf 3/3 Running 0 2d21h
pod/external-dns-7cfc59cdbf-g5tgp 1/1 Running 0 41h
pod/kube-ops-view-9cc4bf44c-wfpbp 1/1 Running 0 2d21h
pod/kube-proxy-5rpv7 1/1 Running 0 2d22h
pod/kube-proxy-78c7d 1/1 Running 0 2d22h
pod/s3-csi-node-b99sn 3/3 Running 0 2d21h
pod/s3-csi-node-l2wj7 3/3 Running 0 2d21h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/aws-load-balancer-webhook-service ClusterIP 172.20.238.252 <none> 443/TCP 2d21h
service/kube-dns ClusterIP 172.20.0.10 <none> 53/UDP,53/TCP 2d22h
service/kube-ops-view LoadBalancer 172.20.93.114 a9de573cf32c240c189921c55595c07e-484039070.ap-northeast-2.elb.amazonaws.com 8080:31263/TCP 2d21h
service/prometheus-stack-kube-prom-coredns ClusterIP None <none> 9153/TCP 19h
service/prometheus-stack-kube-prom-kube-controller-manager ClusterIP None <none> 10257/TCP 19h
service/prometheus-stack-kube-prom-kube-etcd ClusterIP None <none> 2381/TCP 19h
service/prometheus-stack-kube-prom-kube-proxy ClusterIP None <none> 10249/TCP 19h
service/prometheus-stack-kube-prom-kube-scheduler ClusterIP None <none> 10259/TCP 19h
service/prometheus-stack-kube-prom-kubelet ClusterIP None <none> 10250/TCP,10255/TCP,4194/TCP 2d19h
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/aws-node 2 2 2 2 2 <none> 2d22h
daemonset.apps/ebs-csi-node 2 2 2 2 2 kubernetes.io/os=linux 2d21h
daemonset.apps/ebs-csi-node-windows 0 0 0 0 0 kubernetes.io/os=windows 2d21h
daemonset.apps/kube-proxy 2 2 2 2 2 <none> 2d22h
daemonset.apps/s3-csi-node 2 2 2 2 2 kubernetes.io/os=linux 2d21h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/aws-load-balancer-controller 2/2 2 2 2d21h
deployment.apps/coredns 2/2 2 2 2d22h
deployment.apps/ebs-csi-controller 2/2 2 2 2d21h
deployment.apps/external-dns 1/1 1 1 41h
deployment.apps/kube-ops-view 1/1 1 1 2d21h
NAME DESIRED CURRENT READY AGE
replicaset.apps/aws-load-balancer-controller-79574fb866 2 2 2 2d21h
replicaset.apps/coredns-56dfff779f 2 2 2 2d22h
replicaset.apps/ebs-csi-controller-66ccf4755d 2 2 2 2d21h
replicaset.apps/external-dns-7cfc59cdbf 1 1 1 41h
replicaset.apps/kube-ops-view-9cc4bf44c 1 1 1 2d21h
---------------------------------------------------------------------------------
$ kubectl get all -n argocd
NAME READY STATUS RESTARTS AGE
pod/argocd-application-controller-0 1/1 Running 0 2d21h
pod/argocd-applicationset-controller-584f68b9d7-lc94c 1/1 Running 0 2d21h
pod/argocd-dex-server-8577d9498b-8zrd8 1/1 Running 0 2d21h
pod/argocd-notifications-controller-564dcb4995-rm5cs 1/1 Running 0 2d21h
pod/argocd-redis-66d9777b78-x7mk4 1/1 Running 0 2d21h
pod/argocd-repo-server-58c94b5cbf-8s7v8 1/1 Running 0 2d21h
pod/argocd-server-b8bd4f4b5-nfmcf 1/1 Running 0 2d21h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/argocd-applicationset-controller ClusterIP 172.20.189.66 <none> 7000/TCP,8080/TCP 2d21h
service/argocd-dex-server ClusterIP 172.20.5.11 <none> 5556/TCP,5557/TCP,5558/TCP 2d21h
service/argocd-metrics ClusterIP 172.20.116.128 <none> 8082/TCP 2d21h
service/argocd-notifications-controller-metrics ClusterIP 172.20.243.0 <none> 9001/TCP 2d21h
service/argocd-redis ClusterIP 172.20.53.96 <none> 6379/TCP 2d21h
service/argocd-repo-server ClusterIP 172.20.238.149 <none> 8081/TCP,8084/TCP 2d21h
service/argocd-server ClusterIP 172.20.212.138 <none> 80/TCP,443/TCP 2d21h
service/argocd-server-metrics ClusterIP 172.20.31.63 <none> 8083/TCP 2d21h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/argocd-applicationset-controller 1/1 1 1 2d21h
deployment.apps/argocd-dex-server 1/1 1 1 2d21h
deployment.apps/argocd-notifications-controller 1/1 1 1 2d21h
deployment.apps/argocd-redis 1/1 1 1 2d21h
deployment.apps/argocd-repo-server 1/1 1 1 2d21h
deployment.apps/argocd-server 1/1 1 1 2d21h
NAME DESIRED CURRENT READY AGE
replicaset.apps/argocd-applicationset-controller-584f68b9d7 1 1 1 2d21h
replicaset.apps/argocd-dex-server-8577d9498b 1 1 1 2d21h
replicaset.apps/argocd-notifications-controller-564dcb4995 1 1 1 2d21h
replicaset.apps/argocd-redis-66d9777b78 1 1 1 2d21h
replicaset.apps/argocd-repo-server-58c94b5cbf 1 1 1 2d21h
replicaset.apps/argocd-server-b8bd4f4b5 1 1 1 2d21h
NAME READY AGE
statefulset.apps/argocd-application-controller 1/1 2d21h
+1 level=warn ts=2024-08-26T12:59:54.066715957Z caller=klog.go:106 component=k8s_client_runtime func=Warning msg="spec.template.spec.containers[1].ports[0]: duplicate port definition with spec.template.spec.initContainers[0].ports[0]" level=info ts=2024-08-26T12:59:54.067500703Z caller=operator.go:572 component=alertmanager-controller key=monitoring2/prom-kube-prometheus-stack-alertmanager msg="sync alertmanager" level=info ts=2024-08-26T12:59:54.097428357Z caller=operator.go:982 component=prometheusagent-controller msg="we are gonna check if it Matches" level=info ts=2024-08-26T12:59:54.100686912Z caller=operator.go:766 component=prometheus-controller key=monitoring2/prom-kube-prometheus-stack-prometheus msg="sync prometheus"
