[Serve] Allow customize Ray Serve auto scaler scale down logic

[manhld0206]

Description

Now, Ray Serve auto scaler logic is fixed. Allow user to customize the scale down logic based on: worker node, deployment, replica to fit different kinds of business logic.

Use case

For our use case, we want to utilize both on-demand and spot instance nodes. We want to keep a minimum number of replicas always stay on on-demand workers. By customizing the scale down logic, we can ensure this by scaling down replicas on spot instances node first

[ok-scale]

Have you taken a stab at https://docs.ray.io/en/latest/serve/advanced-guides/advanced-autoscaling.html#custom-autoscaling-policies yet?

[manhld0206]

@ok-scale Yes. But it only supports returning the desired number of replicas, not which replica to scale down/terminate

[abrarsheikh]

there is currently an effort to support label selector in serve https://github.com/ray-project/ray/pull/57694. When that happens, maybe this feature can be achieved in the following way

1. on your deployment, set label_selector = on-demand and fallback = spot
2. in your k8s set a maximum limit on number of on-demand instances
3. which means beyond your acceptable threshold, new replicas will be scheduled on spot instances.
4. downscaling would also evict replicas from spot first.

Not a clean API though, but wdyt?

[manhld0206]

@abrarsheikh That's perfect!