Add Scaler that Read Metrics From Current Custom Metrics Adapter

[imroc]

Metric Type and Kubernetes Metric API

There are several metric types when defining HPA: Resource , Pods, Object and External:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: test
spec:
  metrics:
    - type: Pods
       pods:
       ...

• Resource type uses Resource Metrics API(v1beta1.metrics.k8s.io).
• Pods type and Object type uses Custom Metrics API(v1beta1.custom.metrics.k8s.io).
• External type uses External Metrics API(v1beta1.external.metrics.k8s.io).

KEDA occupies the External Metric API, and provide cpu and memory triggers which read metrics from current Resource Metrics API adapter, but no trigger read metrics from current Custom Metrics API adapter.

Scenario: Migrate HPA from Pods or Object metric type

Some cloud vendors provide rich metrics for HPA by default. For example, Tencent TKE provide a lot of HPA metrics for users: https://www.tencentcloud.com/document/product/457/34025

And these metrics are based on Custom Metrics API, which means it has a default adapter of Custom Metrics API, users can define HPA like this:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: test
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: test
  minReplicas: 1
  maxReplicas: 100
  metrics:
    - pods:
        metric:
          name: k8s_pod_rate_cpu_core_used_limit
        target:
          averageValue: "80"
          type: AverageValue
      type: Pods
    - pods:
        metric:
          name: k8s_pod_rate_mem_usage_limit
        target:
          averageValue: "80"
          type: AverageValue
      type: Pods
    - pods:
        metric:
          name: k8s_pod_rate_gpu_used_request
        target:
          averageValue: "60"
          type: AverageValue
      type: Pods

But if users want to use KEDA to add some triggers to the same workload, they need to delete the previously defined HPA because KEDA and HPA cannot be used together, and KEDA didn't provide a trigger that can read metrics from current Custom Mtrics API, so this prevents users from migrating to KEDA.

Proposal: Add Scaler that Read Metrics From Current Custom Metrics API Adapter

The Pods and Object type metric have multiple levels of definitions, and metadata is a map[string]string. It is not possible to directly move existing HPA pods and object metric definitions to metadata. We need to consider how to design it.

[imroc]

Maybe it's better to allow keep existing metrics spec when resue an existing HPA?

For example, add scaledobject.keda.sh/keep-existing-hpa-metrics-spec annotation to ScaledObject:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  annotations:
    scaledobject.keda.sh/keep-existing-hpa-metrics-spec: "true"
spec:
  advanced:
    horizontalPodAutoscalerConfig:
      name: test

Then KEDA only add extra external metric spec to HPA metrics list, keep existing HPA metrics spec in the HPA metrics list.

[imroc]

Maybe it's better to allow keep existing metrics spec when resue an existing HPA?

This is not friendly to GitOps, for example, use ArgoCD to manage YAMLs, and it both include HPA and ScaledObject which reuses HPA, the KEDA will change exsiting HPA's spec, and ArgoCD found that HPA is been changed, then it change it back to original defination.

[imroc]

I think I found the best solution, add a field to horizontalPodAutoscalerConfig, let's say it metrics, we can paste HPA'sspec.metrics to ScaledObject's spec.advanced.horizontalPodAutoscalerConfig.metrics:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: test
  namespace: test
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: test
  pollingInterval: 15
  minReplicaCount: 1
  maxReplicaCount: 100
  advanced:
    horizontalPodAutoscalerConfig:
      metrics:
        - pods:
            metric:
              name: k8s_pod_rate_cpu_core_used_limit
            target:
              averageValue: "80"
              type: AverageValue
          type: Pods
        - pods:
            metric:
              name: k8s_pod_rate_mem_usage_limit
            target:
              averageValue: "80"
              type: AverageValue
          type: Pods
        - pods:
            metric:
              name: k8s_pod_rate_gpu_used_request
            target:
              averageValue: "60"
              type: AverageValue
          type: Pods
  triggers:
    - type: cron
      metadata:
        timezone: Asia/Shanghai
        start: 30 9 * * *
        end: 30 10 * * *
        desiredReplicas: "10"

And KEDA then populate the metrics spec to the HPA that been managed to KEDA, just like keep the behaviour field.

This should be a small but very useful change.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[JorTurFer]

Sorry for the slow response, my life's been a chaos :( I guess that we can include this as another scaler that users can use. I see the potential of this and the use case is really nice as KEDA could work together with other custom metrics provider. At this point (and considering that custom and external metrics have the same metric types) I'm not sure if we should configure a custom metric or just wrap it with an external metric (keeping the control to KEDA for other features like formulas or observability)

@tomkerkhove @zroubalik @dttung2905 @wozniakjan ?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed due to inactivity.

[wozniakjan]

Sorry for the slow response, my life's been a chaos :(

+1, apologies for letting this one through the cracks as well

Add Scaler that Read Metrics From Current Custom Metrics API Adapter I guess that we can include this as another scaler that users can use.

I can see the benefit of this, I'd be happy to review a contribution and help to shape it so it can be merged, but I don't have the capacity currently to implement this

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.