vllm
vllm copied to clipboard
vllm-project
Raise error for long prompt
This is a fix for #113. The program hangs when the input prompt is too long because the check will always succeed and the request will always in the waiting queue and will never be added to the running queue. Add a check before the if statement, if the prompt length is too long, it will raise ValueError directly.
Hi Lily, please check all the possible situations here and make sure you can handle all of them. Thanks!
I move forward with the length check. The original code should already handle the len(prompt) + len(generated) > limit case, so I only deal with model limit and len(prompt) > limit. Let me know if there are any problems, thanks!