vllm-project
[Performance]: V0.6.0 version of the model, benchmarking found that the number of successful responses accounted for half of the number of requests, which is why
Proposal to improve performance
No response
Report of performance regression
No response
Misc discussion on performance
No response
Your current environment (if you think it is necessary)
The output of `python collect_env.py`
Before submitting a new issue...
- [X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
In the v0.6.0 model, the total number of requests is 1000, -- the request-rate is 100, 1) only half of the requests are successfully responded to, why do so many fail? At the same time, check the output length, found that the output length is only 502 non-empty replies, 2) That is to say, the successful response is also None, what is the reason.
@Amber-Believe make sure you follow https://github.com/vllm-project/vllm/issues/8176
@Amber-Believe确保你关注了#8176
Hello, so the meaning here is that the number of successful responses is only half due to the problem of the multi-step scheduler (--num-scheduler-steps 10)?
IIRC vLLM's request buffer size is not large enough to hold 1000 requests. Try use 500 requests instead.
And also would be great if you could use latest vLLM version, there are several stabiliy-related bug fixes there.
This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!