aibrix
aibrix copied to clipboard
Improve the gateway stability for long request
🐛 Describe the bug
I am not sure that's all related to timeout setting. https://github.com/vllm-project/aibrix/pull/879
Steps to Reproduce
send > 20k prompts to the server with 8 * A100, QPS > 1.5 will easily trigger the issue
Expected behavior
gateway should not have any connection issues
Environment
v0.2.1