ray icon indicating copy to clipboard operation
ray copied to clipboard

Add new serve autoscaling parameter `scaling_function`

Open Stack-Attack opened this issue 5 months ago • 5 comments

Why are these changes needed?

Currently, the serve autoscaler makes scaling decisions only based on the most recent Serve Controller computation, even if the serve controller has made many scaling calculations over the scaling delay period. This results in poor autoscaling when clusters utilize long upscale/downscale delays.

Related issue number

https://github.com/ray-project/ray/issues/46497

Checks

  • [ ✓] I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • [ ✓] I've run scripts/format.sh to lint the changes in this PR.
  • [✓ ] I've included any doc changes needed for https://docs.ray.io/en/master/.
    • [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in doc/source/tune/api/ under the corresponding .rst file.
  • [ ✓] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • [✓ ] Unit tests
    • [✓ ] Release tests
    • [ ] This PR is not tested :(

Stack-Attack avatar Sep 27 '24 10:09 Stack-Attack