It doesn't work in EKS

[dominicicerse]

the 'custom-metrics-apiserver' cannot run in EKS: $ kubectl get po -n monitoring NAME READY STATUS RESTARTS AGE custom-metrics-apiserver-55c757b95d-x5zdg 0/1 CrashLoopBackOff 6 9m51s prometheus-8669f7f7d8-slmqq 1/1 Running 0 11m

I checked the log. It has the following error error: Error: cluster doesn't provide client-ca-file

Just past all below: $ kubectl logs custom-metrics-apiserver-55c757b95d-x5zdg -n monitoring I0606 10:15:51.477908 1 round_trippers.go:417] curl -k -v -XGET -H "Accept: application/json, /" -H "User-Agent: adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yaW5nIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImN1c3RvbS1tZXRyaWNzLWFwaXNlcnZlci10b2tlbi1oNXZjYiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjdXN0b20tbWV0cmljcy1hcGlzZXJ2ZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJhNWU0YjQxZC04ODQzLTExZTktODVmMS0wMjdhMzdkNDczMWUiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6bW9uaXRvcmluZzpjdXN0b20tbWV0cmljcy1hcGlzZXJ2ZXIifQ.qzcldE0DaAoaROLHg-mfWY5BegiN9YQwH4qcJGj3-wjWCt9NN3QECLRfdizbFCnglBmc5OmZw8bPc3n7mODBKLRIzqfwryh9RXfVD-heNuB1SnFYp3aTOyDM81k02m2eMcMxULtsfPi1OzyIx_TXqBdxvNiAWgSE9zdtKT9j_DQq2GHAqPLRiSDFI4HepDleyO0CXBGI0xvFiCVuy8J5o7D1aSHlQ232YE4w9ZaB4NgERIRwrQzHZbvGrk6pQn-PEvKFqdHICtR1f8Fgwi3LJtD3NGiichR4zXJNNCtdqxZRM8gJNG7aT-gF_sKH9iYL36EmVef0BRLu4FqSlrBQGw" https://172.20.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication I0606 10:15:51.494442 1 round_trippers.go:436] GET https://172.20.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 16 milliseconds I0606 10:15:51.494464 1 round_trippers.go:442] Response Headers: I0606 10:15:51.494469 1 round_trippers.go:445] Audit-Id: 277f9a0b-bdbe-429f-a08c-2e1e8b6e9231 I0606 10:15:51.494473 1 round_trippers.go:445] Content-Type: application/json I0606 10:15:51.494478 1 round_trippers.go:445] Content-Length: 1648 I0606 10:15:51.494482 1 round_trippers.go:445] Date: Thu, 06 Jun 2019 10:15:51 GMT I0606 10:15:51.494518 1 request.go:836] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"extension-apiserver-authentication","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication","uid":"2e07486e-836e-11e9-85f1-027a37d4731e","resourceVersion":"56","creationTimestamp":"2019-05-31T06:34:45Z"},"data":{"requestheader-allowed-names":"["front-proxy-client"]","requestheader-client-ca-file":"-----BEGIN CERTIFICATE-----\nMIIC0DCCAbigAwIBAgIBADANBgkqhkiG9w0BAQsFADAZMRcwFQYDVQQDEw5mcm9u\ndC1wcm94eS1jYTAeFw0xOTA1MzEwNjM0MDNaFw0yOTA1MjgwNjM0MDNaMBkxFzAV\nBgNVBAMTDmZyb250LXByb3h5LWNhMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB\nCgKCAQEAtpd8oUCa2DYD+O1LSRlaePTLAbrUzdwkxyAssPKNTZgAA3uau4m7HDAW\nAhzpD3JvCohFKTBQ5Dz9kDKesxlQJ9sUVVECuBtskvpNy1GWJPXkDOYRyVDqPmfo\nWDyyGLJwcl34Ztzya5ybcG0nrXXEUElZ+CHvA3jHG30GLAF6kMJHkSFiT943zQml\nnxAsLsb0PYPtrACjYmh3B20Tn3NVuoQ2xPe2lmgjgzY45Ebj9ezjEv0Qli3sdsGC\nW5L+3f/y40vogjSHbgTnhBhedzNhJYco/WFDWxJBvawkKuBuhAhVA3WYxySRffPu\nEdSWK2wqL56cn9vLY0CBU+ZHsoVm1QIDAQABoyMwITAOBgNVHQ8BAf8EBAMCAqQw\nDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAL25JNmj4gvczywkS\nMQ97URsfAYjMCne7TMYxe01bfL+uFnXBRu4DeMA9Q+NQyYmneBiuKJRvhuP0l9U+\nTUWPpiFUBqljQI3azpd0A2E7RWLwUuhcMY37vQxpA+n2D3bfJwx4muQg1RmigX22\n4+tUsmNyzVVAj8H/knk/4oTls5VNSXhMVJ3qyHGmKjUWa+deHHiPT3mBY0S4SP35\nIqL42alcGEju3Vmak4cih848FbgFPaKv2SGpDsaCJNsKWTEs+utAmVuyaCY/iuNj\nyG4wNHpV13GYQ1FLefjCB2uqdAzFD+Wdow6ZvtZXSscnexcmRUNTk+i4j69dtfG4\ntDc0IA==\n-----END CERTIFICATE-----\n","requestheader-extra-headers-prefix":"["X-Remote-Extra-"]","requestheader-group-headers":"["X-Remote-Group"]","requestheader-username-headers":"["X-Remote-User"]"}} Error: cluster doesn't provide client-ca-file Usage: [flags]

Flags: --alsologtostderr log to standard error as well as files --authentication-kubeconfig string kubeconfig file pointing at the 'core' kubernetes server with enough rights to create tokenaccessreviews.authentication.k8s.io. --authentication-skip-lookup If false, the authentication-kubeconfig will be used to lookup missing authentication configuration from the cluster. --authentication-token-webhook-cache-ttl duration The duration to cache responses from the webhook token authenticator. (default 10s) --authorization-kubeconfig string kubeconfig file pointing at the 'core' kubernetes server with enough rights to create subjectaccessreviews.authorization.k8s.io. --authorization-webhook-cache-authorized-ttl duration The duration to cache 'authorized' responses from the webhook authorizer. (default 10s) --authorization-webhook-cache-unauthorized-ttl duration The duration to cache 'unauthorized' responses from the webhook authorizer. (default 10s) --bind-address ip The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank, all interfaces will be used (0.0.0.0). (default 0.0.0.0) --cert-dir string The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored. (default "apiserver.local.config/certificates") --client-ca-file string If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate. --contention-profiling Enable lock contention profiling, if profiling is enabled --discovery-interval duration interval at which to refresh API discovery information (default 10m0s) --enable-swagger-ui Enables swagger ui on the apiserver at /swagger-ui --lister-kubeconfig string kubeconfig file pointing at the 'core' kubernetes server with enough rights to list any described objets --log-flush-frequency duration Maximum number of seconds between log flushes (default 5s) --log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0) --log_dir string If non-empty, write log files in this directory --logtostderr log to standard error instead of files (default true) --metrics-relist-interval duration interval at which to re-list the set of all available metrics from Prometheus (default 10m0s) --profiling Enable profiling via web interface host:port/debug/pprof/ (default true) --prometheus-url string URL and configuration for connecting to Prometheus. Query parameters are used to configure the connection (default "https://localhost") --rate-interval duration period of time used to calculate rate metrics from cumulative metrics (default 5m0s) --requestheader-allowed-names stringSlice List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed. --requestheader-client-ca-file string Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers --requestheader-extra-headers-prefix stringSlice List of request header prefixes to inspect. X-Remote-Extra- is suggested. (default [x-remote-extra-]) --requestheader-group-headers stringSlice List of request headers to inspect for groups. X-Remote-Group is suggested. (default [x-remote-group]) --requestheader-username-headers stringSlice List of request headers to inspect for usernames. X-Remote-User is common. (default [x-remote-user]) --secure-port int The port on which to serve HTTPS with authentication and authorization. If 0, don't serve HTTPS at all. (default 443) --stderrthreshold severity logs at or above this threshold go to stderr (default 2) --tls-ca-file string If set, this certificate authority will used for secure access from Admission Controllers. This must be a valid PEM-encoded CA bundle. Altneratively, the certificate authority can be appended to the certificate provided by --tls-cert-file. --tls-cert-file string File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to /var/run/kubernetes. --tls-private-key-file string File containing the default x509 private key matching --tls-cert-file. --tls-sni-cert-key namedCertKey A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com". (default []) -v, --v Level log level for V logs (default 0) --vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging

panic: cluster doesn't provide client-ca-file

goroutine 1 [running]: main.main() /go/src/github.com/directxman12/k8s-prometheus-adapter/cmd/adapter/adapter.go:41 +0x114

[dominicicerse]

Version: $ kubectl version Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.7", GitCommit:"6f482974b76db3f1e0f5d24605a9d1d38fad9a2b", GitTreeState:"clean", BuildDate:"2019-03-29T16:15:10Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.6-eks-d69f1b", GitCommit:"d69f1bf3669bf00b7f4a758e978e0e7a1e3a68f7", GitTreeState:"clean", BuildDate:"2019-02-28T20:26:10Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

[danielepolencic]

The official docs suggest installing a specific version of the custom metric server. https://docs.aws.amazon.com/eks/latest/userguide/metrics-server.html

Which version are you using?

[dominicicerse]

It should be 0.3.2

Pod Template: Labels: k8s-app=metrics-server Service Account: metrics-server Containers: metrics-server: Image: k8s.gcr.io/metrics-server-amd64:v0.3.2

[yrsurya]

Any way we can use for EKS , when I tried to setup hpa it spinning up max pods( which is set to 10) while initializing as memory is high