Cannot see metrics in AWS cloudwatch, Desired value is always 0.

[akshaysgithub]

Also posted on SO : https://stackoverflow.com/questions/64087291/kubernetes-hpa-hpa-not-showing-information-from-external-metrics

Hello friends,

I am working on setting up memory based metrics, as our app is memory intensive. I have defined a Custom/ContainerInsights namespace, but I still cannot see my metrics in cloudwatch and hpa always shows current value to 0. Only thing I changed since my trial is to migrate hpa from v1beta1 to v2beta2...no luck. What am I missing?

metrics-file :

apiVersion: metrics.aws/v1alpha1
kind: ExternalMetric
metadata:
  name: magentomemory
spec:
  name: magentomemory
  resource:
    resource: "deployment"
  queries:
    - id: magentomemory
      metricStat:
        metric:
          namespace: "Custom/ContainerInsights". ----- > tried only with ContainerInsights as well
          metricName: "pod_memory_utilization"
          dimensions:
            - name: PodName
              value: "magento-prod-deployment"
            - name: ClusterName
              value: "prod-eks"
            - name: Namespace
              value: "default"
        period: 10
        stat: Average
        unit: Percent
      returnData: true

HPA file :

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: resize-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: magento-prod-deployment
  minReplicas: 2
  maxReplicas: 5
  metrics:
  - type: External
    external:
      metricName: magentomemory
      targetValue: 60

kubectl describe externalmetrics magentomemory

kubectl describe externalmetric magentomemory
Name:         magentomemory
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  metrics.aws/v1alpha1
Kind:         ExternalMetric
Metadata:
  Creation Timestamp:  2020-09-27T10:03:12Z
  Generation:          1
  Resource Version:    21363574
  Self Link:           /apis/metrics.aws/v1alpha1/namespaces/default/externalmetrics/magentomemory
  UID:                 a025e5eb-633e-41b7-a04c-cdeabb1fd90a
Spec:
  Name:  magentomemory
  Queries:
    Id:  magentomemory
    Metric Stat:
      Metric:
        Dimensions:
          Name:       PodName
          Value:      magento-prod-deployment
          Name:       ClusterName
          Value:      prod-eks
          Name:       Namespace
          Value:      default
        Metric Name:  pod_memory_utilization
        Namespace:    ContainerInsights
      Period:         10
      Stat:           Average
      Unit:           Percent
    Return Data:      true
  Resource:
    Resource:  deployment
Events:        <none>

In the file below, the current value is always at 60.

kubectl describe hpa resize-hpa
Name:                              resize-hpa
Namespace:                         default
Labels:                            <none>
Annotations:                       <none>
CreationTimestamp:                 Sun, 27 Sep 2020 10:03:34 +0000
Reference:                         Deployment/magento-prod-deployment
Metrics:                           ( current / target )
  "magentomemory" (target value):  0 / 60
Min replicas:                      2
Max replicas:                      5
Deployment pods:                   2 current / 2 desired
Conditions:
  Type            Status  Reason            Message
  ----            ------  ------            -------
  AbleToScale     True    ReadyForNewScale  recommended size matches current size
  ScalingActive   True    ValidMetricFound  the HPA was able to successfully calculate a replica count from external metric magentomemory(nil)
  ScalingLimited  True    TooFewReplicas    the desired replica count is increasing faster than the maximum scale rate
Events:           <none>

Please note, I don't have containerinsights installed, I only installed k8s adapter. I can see it working, logs :

I0927 15:31:26.606499       1 handler.go:67] externalMetricInfo: &{{ } {magentomemory  default /apis/metrics.aws/v1alpha1/namespaces/default/externalmetrics/magentomemory ef450c81-2c82-434f-a3c5-202d21f1f012 21408307 1 2020-09-27 14:55:51 +0000 UTC <nil>  map[] map[] [] []  []} {magentomemory   [{ magentomemory  {{[{PodName magento-prod-deployment} {ClusterName prod-eks} {Namespace default}] pod_memory_utilization Custom/ContainerInsights} 10 Average Percent} 0xc0006cf7fe}]}}
I0927 15:31:26.606598       1 handler.go:68] adding to cache item 'magentomemory' in namespace 'default'
I0927 15:31:26.606608       1 controller.go:122] successfully processed item '{default/magentomemory ExternalMetric}'
I0927 15:31:26.606617       1 controller.go:79] processing next item
I0927 15:31:26.606623       1 controller.go:86] processing item
I0927 15:31:33.009438       1 provider_external.go:18] Received request for namespace: default, metric name: magentomemory, metric selectors: 
I0927 15:31:33.009566       1 client.go:52] using AWS Region: eu-central-1
I0927 15:31:36.158493       1 trace.go:116] Trace[544514530]: "List" url:/apis/external.metrics.k8s.io/v1beta1/namespaces/default/magentomemory,user-agent:kube-controller-manager/v1.15.11 (linux/amd64) kubernetes/065dcec/system:serviceaccount:kube-system:horizontal-pod-autoscaler,client:10.0.188.152 (started: 2020-09-27 15:31:33.009419307 +0000 UTC m=+21636.550396207) (total time: 3.149047824s):
Trace[544514530]: [3.148993074s] [3.148984148s] Listing from storage done
I0927 15:31:48.462973       1 provider_external.go:18] Received request for namespace: default, metric name: magentomemory, metric selectors: 
I0927 15:31:48.463088       1 client.go:52] using AWS Region: eu-central-1
I0927 15:31:51.615349       1 trace.go:116] Trace[335835293]: "List" url:/apis/external.metrics.k8s.io/v1beta1/namespaces/default/magentomemory,user-agent:kube-controller-manager/v1.15.11 (linux/amd64) kubernetes/065dcec/system:serviceaccount:kube-system:horizontal-pod-autoscaler,client:10.0.188.152 (started: 2020-09-27 15:31:48.462954873 +0000 UTC m=+21652.003931790) (total time: 3.152369505s):
Trace[335835293]: [3.152318739s] [3.15231121s] Listing from storage done

[devoprock]

@akshaysgithub ,I am getting same issue,Did you resloved your issue ?

[toyama0919]

I have the same problem. custom metrics is not working.

[jack1902]

@akshaysgithub so once i had the IRSA annotation correct and was able to use the role (i sniped the pod to ensure it wasn't using a bad token or anything). The thing that fixed it for me was the type (I originally had None specified) but when i moved to Count it started to populate as expected

[misteio]

You can take a look there : https://github.com/awslabs/k8s-cloudwatch-adapter/issues/51 They explain how you can debug this case.

After lot of research, I was stuck like you. And I realize that when I want to retrieve my custom metrics (Generated by a SaaS), I needed to specify all dimensions, that they have added. And then suddently, all my metrics appears. Link to AWS Doc

[kjdeepthi]

I'm hitting the same issue for unit: Percent with hpa v2beta2, on the other hand unit: Count works fine