not sure how to install the adapter

[Arnold1]

Expected Behavior

Actual Behavior

im not sure how to install the adapter. make was working fine. but when I start the kube-metrics-adapter binary I get some error. is there more doc somewhere, which I missed to read?

$ make

$ KUBERNETES_SERVICE_HOST=100.63.0.10 KUBERNETES_SERVICE_PORT=443 ./kube-metrics-adapter

panic: failed to get delegated authentication kubeconfig: failed to get delegated authentication kubeconfig: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory

goroutine 1 [running]:
main.main()
	/Users/geri/Work/cmv-tfserving/provisioning/zalando-kube-metrics-adapter/kube-metrics-adapter/main.go:40 +0x10e

is there an official helm chart to install the adapter - which works with aws Kops? what I found is this: https://hub.helm.sh/charts/banzaicloud-stable/kube-metrics-adapter can I use this helm chart for install the zalando metrics-server?

currently I have this metrics-server installed, will I need to deinstall it to use zalando metrics-server?

$ helm ls
NAME          	REVISION	UPDATED                 	STATUS  	CHART               	APP VERSION	NAMESPACE   
istio         	2       	Fri Dec 27 18:33:51 2019	DEPLOYED	istio-1.4.0         	1.4.0      	istio-system
kube2iam      	1       	Mon Dec 16 16:36:50 2019	DEPLOYED	kube2iam-2.1.0      	0.10.7     	kube-system 
metrics-server	1       	Mon Dec 16 14:55:56 2019	DEPLOYED	metrics-server-2.8.8	0.3.5      	kube-system 

Steps to Reproduce the Problem

Specifications

• Version:

git clone https://zalando-incubator/kube-metrics-adapter
commit 4412e3dca486658a04bc2585e1843c170da85e21 (HEAD -> master, origin/master, origin/HEAD)

• Platform: I locally have Mac osx

• Subsystem: aws kops with k8s

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"7015f71e75f670eb9e7ebd4b5749639d42e20079", GitTreeState:"clean", BuildDate:"2019-11-13T11:11:50Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

$ helm version
Client: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}

[mikkeloscar]

Hi,

Running the adapter locally is not so helpful as it needs to be in a cluster to be useful.

We provide all the needed manifests for deploying here: https://github.com/zalando-incubator/kube-metrics-adapter/tree/master/docs just apply them one by one.

We currently don't provide a helm chart, and I'm afraid that the https://hub.helm.sh/charts/banzaicloud-stable/kube-metrics-adapter chart may be out-of-date.

I'll add a quick section to the README, so it's more clear how to deploy it.

[Arnold1]

@mikkeloscar thanks a lot. does it work when I deploy the manifests on AWS Kops? also I need to change the aws region accordingly? https://github.com/zalando-incubator/kube-metrics-adapter/blob/master/docs/deployment.yaml#L32-L33

ok how can I use this adapter do hpa by latency rather than by cpu? but I want to change from cpu scaling to scale by maximum req duration P99 (milliseconds). if the pod's P99 is > 500ms I want to add another pod. could you show it and update the following template?

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  labels:
    app: tf-test
    app.kubernetes.io/name: tf-test
    model-partition: "88-2"
  name: tf-test-model-88-2
spec:
  maxReplicas: 88
  minReplicas: 2
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: "tf-test-model-88-2-0"
  targetCPUUtilizationPercentage: 60

[mikkeloscar]

The region is only relevant for the SQS metrics, if you don't need it you can leave it out I believe.

If you want to scale on something like p99 then you need a way to collect those metrics, prometheus would be an obvious choice. If you have prometheus setup you can configure an HPA like shown in this example: https://github.com/zalando-incubator/kube-metrics-adapter#example-external-metric

Note that your kube-metrics-adapter needs to be configured with the prometheus endpoint. Here's an example from how we run it: https://github.com/zalando-incubator/kubernetes-on-aws/blob/6d7835a5906847837ec905e43573eb8b9e8dc0f6/cluster/manifests/kube-metrics-adapter/deployment.yaml#L38

[sancyx]

Hi @Arnold1, we've upgraded the helm chart in our repo, https://hub.helm.sh/charts/banzaicloud-stable/kube-metrics-adapter/0.1.1 so you can use it if you want.

[Miniland1333]

Hi,

Running the adapter locally is not so helpful as it needs to be in a cluster to be useful.

We provide all the needed manifests for deploying here: https://github.com/zalando-incubator/kube-metrics-adapter/tree/master/docs just apply them one by one.

We currently don't provide a helm chart, and I'm afraid that the https://hub.helm.sh/charts/banzaicloud-stable/kube-metrics-adapter chart may be out-of-date.

I'll add a quick section to the README, so it's more clear how to deploy it.

Hello @mikkeloscar, were you ever able to add that section to the README? I actually had to come here to figure out how to do it as a newbie.

[AnthonMS]

Hello, and sorry for commenting on an old issue. But I am new to this k8s world and I'm just trying to get the HPA to work properly for the PHP application at my work. scaling based on CPU is simply too slow, and it will be scaled after all our users leave again because they have waited 10 seconds for a response.

How do I get this to run in our GKE instance? I know how k8s works in general and normally how to add an adaptor. But how do I add this adapter to an existing deployment? Which files in https://github.com/zalando-incubator/kube-metrics-adapter/tree/master/docs do I need to apply and in what order?

I'm sorry for these noob questions. I'm just trying to learn and understand. And learning k8s on GKE hasn't been the easiest with all these differences.

[szuecs]

@AnthonMS in general apply CRD and RBAC first, but keep in mind this is a "cluster service", it's not for only a single application. The hpa object can then have "more" features with kube-metrics-adapter running. You can always check our config in https://github.com/zalando-incubator/kubernetes-on-aws/tree/dev/cluster/manifests/kube-metrics-adapter, too, but it's templated, so you need to change some stuff to make it GKE ready

[AnthonMS]

@szuecs thank you for the quick reply.

I have tried setting up the adapter by applying all the yaml files from docs dir. And I started out by applying CRD and RBAC first.

I do not seem to get any errors, but the HPA I create with custom metrics are all failing. Under targets it says Unknown/1k. I assume that's because the custom metric isn't working.

I have tried all the different ports that makes sense in our setup. 8000, 8080, 9000, 9090, 80. None of them gives a different result under target. I do not understand what it is I'm doing wrong, but I just haven't found a way to get custom metrics without having a ton of knowledge about k8s. I'm trying to build this knowledge, but nothing works... When I run the app with just the docker, then I access it through port 8000. The NodePort nginx service is forwarding port 80 to 8080.

I have no idea what I'm doing. Is there any help at all out there, that's up to date and tells how I get custom metrics in a GKE cluster??

[szuecs]

@AnthonMS maybe ask GKE support, if it's possible to run kube-metrics-adapter in GKE. I am not sure if all requirements are made. One problem could be your kubernetes version and the external-metrics-apiservice.yaml:

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.external.metrics.k8s.io

Maybe in your kubernetes version it's not v1beta1 but v1, for example.

[AnthonMS]

@szuecs It seems to supports it from what I have read. I have also added a comment on issue #254 since I am getting the same errors. I have just ran the logs command as he did, and I get the same error that connection was refused on ip:port/metrics. I am running the same setup with a Load Balancer forwarding port 80 to 8080. So I'm assuming that is the port I need to put in the hpa?

Do I need to host a /metrics myself that has the metrics I would like to scale based on? I thought this adapter made it possible to scale based on request_pet_second to a pod in itself? I can get the hipages/php-fpm_exporter container to export fpm metrics to /metrics, but I have removed this container as of now because I couldn't get those metrics to be available to k8s. Do I need another container to export pod metrics?

[szuecs]

Requests per second is possible if you run skipper as ingress controller https://opensource.zalando.com/skipper/kubernetes/ingress-controller/