custom autoscaler not working

[manishrajkarnikar]

✗ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/services/sample-metrics-app/http_requests | jq
{
  "kind": "MetricValueList",
  "apiVersion": "custom.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/services/sample-metrics-app/http_requests"
  },
  "items": [
    {
      "describedObject": {
        "kind": "Service",
        "name": "sample-metrics-app",
        "apiVersion": "/__internal"
      },
      "metricName": "http_requests",
      "timestamp": "2017-11-29T21:31:13Z",
      "value": "539m"
    }
  ]
}

Note that http_requests has value of "539m". It should be a number and not memory. In my case I was expecting 120 (that was my transaction per second i had set via my load test tool). Due to this issue hpa does not scale the replicaset appropriately. Prometheus metrics was verified to show right value.

I1129 21:27:07.806751       1 api.go:74] GET http://sample-metrics-prom.default.svc:9090/api/v1/query?query=sum%28rate%28http_requests_total%7Bservice%3D%22sample-metrics-app%22%2Cnamespace%3D%22default%22%7D%5B
1m%5D%29%29+by+%28service%29&time=1511990827.805 200 OK
I1129 21:27:07.806847       1 api.go:93] Response Body: {"status":"success","data":{"resultType":"vector","result":[{"metric":{"service":"sample-metrics-app"},"value":[1511990827.805,"30.54"]}]}}

This is what describing the hpa shows.

➜  kubectl describe hpa sample-metrics-app-hpa
Name:                                             sample-metrics-app-hpa
Namespace:                                        default
Labels:                                           <none>
Annotations:                                      kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"sample-metrics-app-hpa","namespace":"default"...
CreationTimestamp:                                Wed, 29 Nov 2017 21:29:58 +0000
Reference:                                        Deployment/sample-metrics-app
Metrics:                                          ( current / target )
  "http_requests" on Service/sample-metrics-app:  125300m / 100
Min replicas:                                     1
Max replicas:                                     10
Conditions:
  Type            Status  Reason            Message
  ----            ------  ------            -------
  AbleToScale     True    ReadyForNewScale  the last scale time was sufficiently old as to warrant a new scale
  ScalingActive   True    ValidMetricFound  the HPA was able to succesfully calculate a replica count from Service metric http_requests
  ScalingLimited  True    TooManyReplicas   the desired replica count was more than the maximum replica count
Events:
  Type     Reason                        Age              From                       Message
  ----     ------                        ----             ----                       -------
  Warning  FailedGetObjectMetric         8m (x9 over 2h)  horizontal-pod-autoscaler  unable to get metric http_requests: Service on default sample-metrics-app/unable to fetch metrics from API: the server could not find the requested resource (get services.custom.metrics.k8s.io sample-metrics-app)
  Warning  FailedComputeMetricsReplicas  8m (x9 over 2h)  horizontal-pod-autoscaler  failed to get object metric value: unable to get metric http_requests: Service on default sample-metrics-app/unable to fetch metrics from API: the server could not find the requested resource (get services.custom.metrics.k8s.io sample-metrics-app)

[zh168654]

i am running into the same problem, have u solved it?

[derekperkins]

I've got the same issue

[huxiaoliang]

I got same issue as well:

[root@kvm-014724 autoscaling]# kubectl describe hpa
Name:                                             sample-metrics-app-hpa
Namespace:                                        default
Labels:                                           <none>
Annotations:                                      <none>
CreationTimestamp:                                Wed, 28 Feb 2018 12:07:33 +0800
Reference:                                        Deployment/sample-metrics-app
Metrics:                                          ( current / target )
  "http_requests" on Service/sample-metrics-app:  <unknown> / 100
Min replicas:                                     2
Max replicas:                                     10
Conditions:
  Type           Status  Reason                 Message
  ----           ------  ------                 -------
  AbleToScale    True    SucceededGetScale      the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetObjectMetric  the HPA was unable to compute the replica count: unable to get metric http_requests: Service on default sample-metrics-app/unable to fetch metrics from API: the server could not find the requested resource (get services.custom.metrics.k8s.io sample-metrics-app)
Events:
  Type     Reason                        Age                 From                       Message
  ----     ------                        ----                ----                       -------
  Warning  FailedGetObjectMetric         7m (x101 over 57m)  horizontal-pod-autoscaler  unable to get metric http_requests: Service on default sample-metrics-app/unable to fetch metrics from API: the server could not find the requested resource (get services.custom.metrics.k8s.io sample-metrics-app)
  Warning  FailedComputeMetricsReplicas  2m (x111 over 57m)  horizontal-pod-autoscaler  failed to get object metric value: unable to get metric http_requests: Service on default sample-metrics-app/unable to fetch metrics from API: the server could not find the requested resource (get services.custom.metrics.k8s.io sample-metrics-app)

[huxiaoliang]

@luxas could you help double check it, thanks!

[zihaoyu]

I noticed that the sample server outputs metric http_requests_total (https://github.com/luxas/kubeadm-workshop/blob/master/images/autoscaling/server.js#L12), while the HPA monitors http_requests (https://github.com/luxas/kubeadm-workshop/blob/master/demos/monitoring/sample-metrics-app.yaml#L95) Not sure if that’s intentional. I’ve tried both metric names in my HPA and no luck

[rickslick]

@zihaoyu @huxiaoliang I have found a work around. As @zihaoyu mentioned the server app is giving out http_requests_total while the prometheus adapter is looking out for http_request(figuring out why ) , So what we did is that we modified the application (https://github.com/luxas/kubeadm-workshop/blob/master/images/autoscaling/server.js ) to use http_request and redeployed the container and autoscaling started working

[thejasbabu]

@luxas Can you tell how the http_requests in hpa is getting resolved? I have looked at the default rules in DirectXMan12/k8s-prometheus-adapter, there seems like no default rule for this.