vllm-project
Fix #issues/320
try: async for request_output in results_generator: if await request.is_disconnected():
The await request.is_disconnected is never excueted if something error happed(maybe length over max) ,so the request never quitted which cause the schedule to stuck
@metacryptom Thanks for your contribution! Please correct me if I'm wrong. If this is merely for fixing the ValueError that happened when the input is too long, #273 is actually fixing it. If this is for fixing for the case when a client disconnects during streaming, I believe FastAPI automatically shut down the coroutine and calls the following background task to abort the request:
https://github.com/vllm-project/vllm/blob/598dc4b79a0078fe14e0c134d0b53b4842a8b227/vllm/entrypoints/api_server.py#L50-L52
If both these two cases can be effectively handled as what I understand, is this PR still necessary?
Not just the case the input is too long, when the request can't be executed and added to swap queue ,the new coming request can't be executed either . I think the main problems here are that when in none stream mode (the code you paste above only works in stream mode), the disconnect check can't code can't be executed before the generator yield some outputs ,this code will cause potential issues ,the long input is just one of them. The better way is to check disconnected on the generator loop .
yes, i find some special tokens decode would also cause stucked. All other requests are blocked.
Close this PR since we did a major refactor of async_llm_engine.py recently and this issue should have been fixed. Feel free to open a new PR if this issue persists.