[Autoscaler v2] Autoscaler submits scale request to kuberay operator

[Chong-Li]

Description

For now, autoscaler depends on KubernetesHttpApiClient to submit scale request to k8s api server. However, k8s api server is generally rate limited while requiring strict authorization. So we may need a new http client (say KubeRayHttpApiClient) which directly talks to kuberay operator. The REST format might look this way:

• GET

  • url to get head pod resource version: http://{KUBERAY_OPERATOR_ADDRESS}/api/v1/namespaces/{K8S_NAMESPACE}/pods/{RAY_HEAD_POD_NAME}

  • url to get rayclusters: http://{KUBERAY_OPERATOR_ADDRESS}/apis/ray.io/v1alpha1/namespaces/{K8S_NAMESPACE}/rayclusters/{CLUSTER_NAME}

  • url to get pods: http://{KUBERAY_OPERATOR_ADDRESS}/api/v1/namespaces/{K8S_NAMESPACE}/pods?labelSelector=ray.io/cluster={CLUSTER_NAME}

  • headers (without k8s token):

{
   "AccessKey":"KubeRayOperator",
   "AccessSecret":"S3ViZVJheU9wZXJhdG9y", // base64 encode of the access key
   "Content-Type":"application/json"
}

• PATCH
  • url: http://{KUBERAY_OPERATOR_ADDRESS}/apis/ray.io/v1alpha1/namespaces/{K8S_NAMESPACE}/rayclusters/{CLUSTER_NAME}
  • headers: as above
  • example payload:

[
   {
       "op": "replace",
       "path": "/spec/workerGroupSpecs/3/replicas",
       "value": 5,
   },
   {
       "op": "replace",
       "path": "/spec/workerGroupSpecs/0/scaleStrategy",
       "value": {
           "workersToDelete": [
               "raycluster-autoscaler-worker-small-group-dkz2r"
           ]
       },
   },
]

@Qstar any comments?

Use case

No response