ray icon indicating copy to clipboard operation
ray copied to clipboard

[Serve] Support callables in `serve.batch` args

Open Yard1 opened this issue 2 years ago • 1 comments

Why are these changes needed?

This PR allows for callables to be used in serve.batch arguments, allowing users to add logic to dynamically change the max batch size and timeout. It also makes batch_queue_cls configurable as an advanced API.

Related issue number

Checks

  • [ ] I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • [ ] I've run scripts/format.sh to lint the changes in this PR.
  • [ ] I've included any doc changes needed for https://docs.ray.io/en/master/.
    • [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in doc/source/tune/api/ under the corresponding .rst file.
  • [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • [ ] Unit tests
    • [ ] Release tests
    • [ ] This PR is not tested :(

Yard1 avatar Jun 13 '23 19:06 Yard1

@Yard1 would you mind adding a short extension to https://docs.ray.io/en/latest/serve/tutorials/batch.html showing how to do this?

edoakes avatar Jun 13 '23 19:06 edoakes

As discussed offline, let's go with "magic" instance variable names for now:

  • __ray_serve_max_batch_size
  • __ray_serve_batch_wait_timeout_s

edoakes avatar Jun 15 '23 19:06 edoakes