vllm-project
upstream connect error or disconnect/reset before headers
🐛 Describe the bug
Can we list potential reasons for connection issues. In recently benchmark testing, I see a few similar cases.
Steps to Reproduce
Run benchmark, seems reproducible if the backend server sometimes is done or not respondingly. for example, the kv cache server is done.
Expected behavior
it should return readable http code or messages for 503.
Environment
- AIBrix v0.3.0-rc.1
- Benchmark
This error occurs when envoy proxy tries to forward request and no pod is in ready state. When this error occurs envoy proxy does not receives response (headers or body), therefore envoy proxy directly returns response to client without interaction with gateway plugin which can provide custom or user friendly response.
I had a chat with envoy gateway committers and this is expected behavior, they also let me know to explore Direct Response for specific error code in this case 503. I need to explore more because it will coincide with 503 error code generated by gateway-plugin.
