crane icon indicating copy to clipboard operation
crane copied to clipboard
verified publisher icon gocrane

通过crand与原生prometheus-adapter实现ehpa扩缩功能

Open saikey0379 opened this issue 3 years ago • 1 comments

What type of PR is this?

通过crond的metric采集ehpa指标，解耦metric-adapter，实现与原生prometheus-adapter的兼容

使用示例

APIservice调整

将external的apiservice调整为原生的prometheus-adapter

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.external.metrics.k8s.io
spec:
  group: external.metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: prometheus-adapter
    namespace: default
    port: 443
  version: v1beta1
  versionPriority: 100

craned指标查询

# curl -sL 10.244.0.134:8080/metrics | grep ^crane | grep -E "cron|predi"| grep -v ^#
crane_cron_external{resourceIdentifier="cron",targetKind="Deployment",targetName="project-test",targetNamespace="test"} 7 1660202258972
crane_prediction_time_series_prediction_external{algorithm="dsp",resourceIdentifier="server_picker",targetKind="Deployment",targetName="project-test",targetNamespace="test",type="ExpressionQuery"} 124.85 1660202265000
crane_prediction_time_series_prediction_external_with_window{algorithm="dsp",resourceIdentifier="server_picker",targetKind="Deployment",targetName="project-test",targetNamespace="test",type="ExpressionQuery"} 140.5433349609375 1660202258972
crane_prediction_time_series_prediction_resource{algorithm="dsp",resource="cpu",resourceIdentifier="cpu",targetKind="Deployment",targetName="project-test",targetNamespace="test",type="ResourceQuery"} 3.79116 1660202265000
crane_prediction_time_series_prediction_resource_with_window{algorithm="dsp",resource="cpu",resourceIdentifier="cpu",targetKind="Deployment",targetName="project-test",targetNamespace="test",type="ResourceQuery"} 4.085559844970703 1660202258972

注：server_picker为原始external指标

  • crane_cron_external定义cron类型定时扩容指标
  • crane_prediction_time_series_prediction_resource_with_window定义resource类型预测指标
  • crane_prediction_time_series_prediction_external_with_window定义external类型预测指标

prometheus-adapter配置示例

apiVersion: v1
data:
  config.yaml: |
    externalRules:
    - seriesQuery: '{__name__="server_picker",pod_name!=""}'
      metricsQuery: 'sum by(node, route)(rate(<<.Series>>{<<.LabelMatchers>>}[1m]))'
      resources:
        namespaced: false
    - seriesQuery: '{__name__="crane_cron_external",pod_name!=""}'
      metricsQuery: 'max(<<.Series>>{<<.LabelMatchers>>})'
      resources:
        namespaced: false
    - seriesQuery: '{__name__="crane_prediction_time_series_prediction_resource_with_window",pod_name!=""}'
      metricsQuery: 'max(<<.Series>>{<<.LabelMatchers>>})'
      resources:
        namespaced: false
    - seriesQuery: '{__name__="crane_prediction_time_series_prediction_external_with_window",pod_name!=""}'
      metricsQuery: 'max(<<.Series>>{<<.LabelMatchers>>})'
      resources:
        namespaced: false

EHPA配置示例

apiVersion: autoscaling.crane.io/v1alpha1
kind: EffectiveHorizontalPodAutoscaler
metadata:
  name: project-test
  namespace: test
  annotations:
    metric-query.autoscaling.crane.io/server_picker: |
      sum by(node, route)(rate(server_picker{node="test",name="server_picker"}[1m]))
spec:
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 660
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15
      - type: Pods
        value: 10
        periodSeconds: 15
      selectPolicy: Max
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: project-test
  minReplicas: 3
  maxReplicas: 25
  scaleStrategy: Preview
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
  - type: External
    external:
      metric:
        name: server_picker
        selector:
          matchLabels:
            node: "test"
            name: "server_picker"
      target:
        averageValue: 10
        type: AverageValue
  crons:
  - name: "cron1"
    description: "scale up"
    start: "0 * * * *"
    end: "5 * * * *"
    targetReplicas: 4
  prediction:
    predictionWindowSeconds: 600
    predictionAlgorithm:
      algorithmType: dsp
      dsp:
        estimators:
          fft:
            - marginFraction: "0.1"
              minNumOfSpectrumItems: 2
              lowAmplitudeThreshold: "0.2"
              highFrequencyThreshold: "6.0"
        sampleInterval: "15s"
        historyLength: "16d"

应用该yaml后，将自动生成以ehpa前缀的hpa以及tsp[ehpa-project-test] 注:annotations部分需与metric.External.Name相同，改部分作为tsp模型的数据源，需与prometheus-adapter中对应指标语句相同 此配置示例中开启了cron以及prediction，prediction部分metric配置将生成相应tsp指标

TSP查看

kubectl -n test get tsp ehpa-project-test -o yaml

apiVersion: prediction.crane.io/v1alpha1
kind: TimeSeriesPrediction
metadata:
spec:
  predictionMetrics:
  - algorithm:
    resourceIdentifier: cpu
    resourceQuery: cpu
    type: ResourceQuery
  - algorithm:
    expressionQuery:
      expression: |
        sum by(node, route)(rate(erver_picker{node="test",name="server_picker"}[1m]))
    resourceIdentifier: server_picker
    type: ExpressionQuery
  predictionWindowSeconds: 600

resourceIdentifier值为ehpa中相应metric.Name 注：该值将作为预测指标的matchLabel，实现预测指标匹配

查看hpa状态

# kubectl -n test describe hpa ehpa-project-test
Name:                                                                                     ehpa-project-test
Namespace:                                                                                test
Labels:                                                                                   app.kubernetes.io/managed-by=effective-hpa-controller
                                                                                          autoscaling.crane.io/effective-hpa-uid=c3783ed1-0d0e-45f8-af52-94c0771f766c
Annotations:                                                                              <none>
CreationTimestamp:                                                                        Thu, 11 Aug 2022 15:07:23 +0800
Reference:                                                                                Substitute/ehpa-project-test
Metrics:                                                                                  ( current / target )
  "server_picker" (target average value):                                                 5251m / 10
  "crane_prediction_time_series_prediction_resource_with_window" (target average value):  162m / 300m
  "crane_prediction_time_series_prediction_external_with_window" (target average value):  5715m / 10
  "crane_cron_external" (target average value):                                           200m / 1
  resource cpu on pods  (as a percentage of request):                                     29% (147m) / 60%
Min replicas:                                                                             3
Max replicas:                                                                             25

此时可以看到hpa共有五个指标

  • "server_picker" #基础外部指标
  • "crane_prediction_time_series_prediction_resource_with_window" #resource类型预测指标
  • "crane_prediction_time_series_prediction_external_with_window" #external类型预测指标
  • "crane_cron_external" #cron计划任务指标
  • resource cpu on pods #基础cpu指标

查看hpa配置

# kubectl get HorizontalPodAutoscaler.v2beta2.autoscaling -ntest ehpa-project-test -o yaml

相应指标如下，通过prometheus-adapter获取

  - external:
      metric:
        name: crane_prediction_time_series_prediction_resource_with_window
        selector:
          matchLabels:
            resourceIdentifier: cpu
            targetKind: Deployment
            targetName: project-test
            targetNamespace: test
      target:
        averageValue: 300m
        type: AverageValue
    type: External
  - external:
      metric:
        name: crane_prediction_time_series_prediction_external_with_window
        selector:
          matchLabels:
            resourceIdentifier: server_picker
            targetKind: Deployment
            targetName: project-test
            targetNamespace: test
      target:
        averageValue: "10"
        type: AverageValue
    type: External
  - external:
      metric:
        name: crane_cron_external
        selector:
          matchLabels:
            resourceIdentifier: cron
            targetKind: Deployment
            targetName: project-test
            targetNamespace: test
      target:
        averageValue: "1"
        type: AverageValue
    type: External

结束

saikey0379 avatar Aug 11 '22 07:08 saikey0379

🎉 Successfully Build Images. Now Support ARM Platforms. Comment Post Time: 2022-08-16 15:56 Git Version: 55b0802

Docker Registry

Overview: https://hub.docker.com/u/gocrane

Image Pull Command
crane-agent:pr-469-55b0802 docker pull gocrane/crane-agent:pr-469-55b0802
dashboard:pr-469-55b0802 docker pull gocrane/dashboard:pr-469-55b0802
metric-adapter:pr-469-55b0802 docker pull gocrane/metric-adapter:pr-469-55b0802
craned:pr-469-55b0802 docker pull gocrane/craned:pr-469-55b0802

Quick Deploy - Helm

helm repo add crane https://finops-helm.pkg.coding.net/gocrane/gocrane
helm install crane -n crane-system --create-namespace \
                   --set craned.image.repository=gocrane/craned \
                   --set craned.image.tag=pr-469-55b0802 \
                   --set metricAdapter.image.repository=gocrane/metric-adapter \
                   --set metricAdapter.image.tag=pr-469-55b0802 \
                   --set craneAgent.image.repository=gocrane/crane-agent \
                   --set craneAgent.image.tag=pr-469-55b0802 \
                   --set cranedDashboard.image.repository=gocrane/dashboard \
                   --set cranedDashboard.image.tag=pr-469-55b0802 crane/crane

Coding Registry

Overview: https://finops.coding.net/public-artifacts/gocrane/crane/packages

Image Pull Command
crane-agent:pr-469-55b0802 docker pull finops-docker.pkg.coding.net/gocrane/crane/crane-agent:pr-469-55b0802
dashboard:pr-469-55b0802 docker pull finops-docker.pkg.coding.net/gocrane/crane/dashboard:pr-469-55b0802
metric-adapter:pr-469-55b0802 docker pull finops-docker.pkg.coding.net/gocrane/crane/metric-adapter:pr-469-55b0802
craned:pr-469-55b0802 docker pull finops-docker.pkg.coding.net/gocrane/crane/craned:pr-469-55b0802

Quick Deploy - Helm

helm repo add crane https://finops-helm.pkg.coding.net/gocrane/gocrane
helm install crane -n crane-system --create-namespace \
                   --set craned.image.repository=finops-docker.pkg.coding.net/gocrane/crane/craned \
                   --set craned.image.tag=pr-469-55b0802 \
                   --set metricAdapter.image.repository=finops-docker.pkg.coding.net/gocrane/crane/metric-adapter \
                   --set metricAdapter.image.tag=pr-469-55b0802 \
                   --set craneAgent.image.repository=finops-docker.pkg.coding.net/gocrane/crane/crane-agent \
                   --set craneAgent.image.tag=pr-469-55b0802 \
                   --set cranedDashboard.image.repository=finops-docker.pkg.coding.net/gocrane/crane/dashboard \
                   --set cranedDashboard.image.tag=pr-469-55b0802 crane/crane

Ghcr Registry

Overview: https://github.com/orgs/gocrane/packages?repo_name=crane

Image Pull Command
crane-agent:pr-469-55b0802 docker pull ghcr.io/gocrane/crane/crane-agent:pr-469-55b0802
dashboard:pr-469-55b0802 docker pull ghcr.io/gocrane/crane/dashboard:pr-469-55b0802
metric-adapter:pr-469-55b0802 docker pull ghcr.io/gocrane/crane/metric-adapter:pr-469-55b0802
craned:pr-469-55b0802 docker pull ghcr.io/gocrane/crane/craned:pr-469-55b0802

Quick Deploy - Helm

helm repo add crane https://finops-helm.pkg.coding.net/gocrane/gocrane
helm install crane -n crane-system --create-namespace \
                   --set craned.image.repository=ghcr.io/gocrane/crane/craned \
                   --set craned.image.tag=pr-469-55b0802 \
                   --set metricAdapter.image.repository=ghcr.io/gocrane/crane/metric-adapter \
                   --set metricAdapter.image.tag=pr-469-55b0802 \
                   --set craneAgent.image.repository=ghcr.io/gocrane/crane/crane-agent \
                   --set craneAgent.image.tag=pr-469-55b0802 \
                   --set cranedDashboard.image.repository=ghcr.io/gocrane/crane/dashboard \
                   --set cranedDashboard.image.tag=pr-469-55b0802 crane/crane

github-actions[bot] avatar Aug 11 '22 08:08 github-actions[bot]

#issue_id：https://github.com/gocrane/crane/issues/474

saikey0379 avatar Aug 16 '22 07:08 saikey0379