ray
ray copied to clipboard
[serve] vllm example to serve llm models
Adds a documentation example using vLLM to serve LLM models on Ray Serve.
This is a copy of https://github.com/ray-project/ray/pull/45325 + add a build environment for ray serve + vllm.
Test:
- CI
@akshay-anyscale, @edoakes i managed to create an environment for the test to run but it fails for some other reasons https://buildkite.com/ray-project/microcheck/builds/237#018f8c35-e5a1-443d-8cf9-bbb481af6c1e/177-2429; if this makes sense feel free to change this pr, thankkks
@aslonnie intended to get merged, but will need serve folks to pick up and finish the job ;)
@akshay-anyscale, @edoakes i managed to create an environment for the test to run but it fails for some other reasons https://buildkite.com/ray-project/microcheck/builds/237#018f8c35-e5a1-443d-8cf9-bbb481af6c1e/177-2429; if this makes sense feel free to change this pr, thankkks
Pushed a commit to change the dtype, hopefully that fixes things.
Is ray-llm going to be deprecated and this example will be the recommended way to run vllm on Ray?