amazon-eks-pod-identity-webhook icon indicating copy to clipboard operation
amazon-eks-pod-identity-webhook copied to clipboard

Published 20 hours ago • verified publisher icon aws

Prometheus scraper misconfigured in the k8s service

Open frdeboffles opened this issue 3 years ago • 0 comments

What happened: Getting plenty of errors like:

2022/01/21 00:39:53 http: TLS handshake error from 10.97.36.76:42820: remote error: tls: bad certificate

In my cluster, this IP relates to Prometheus server. Looking at the service configuration I found that the Prometheus scraper annotations were pointing at the webhook TLS port instead of the metrics port.

What you expected to happen: No errors in the logs

How to reproduce it (as minimally and precisely as possible): Install Prometheus in the cluster where the webhook is running with scrape enabled for service endpoints.

Anything else we need to know?:

Environment:

  • AWS Region: us-east-1
  • EKS Platform version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.platformVersion): eks4
  • Kubernetes version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.version): 1.21
  • Webhook Version: master branch at a65cc3d9c61cf6fc43f0f985818c474e0867d786

frdeboffles avatar Jan 21 '22 01:01 frdeboffles