FastChat
FastChat copied to clipboard
Fix Semaphore release Issue in `api_generate_stream` Function of vllm_worker
trafficstars
Why are these changes needed?
This PR addresses a potential issue in the api_generate_stream function where the semaphore might not be properly released in all scenarios. Specifically, if an exception occurs during the execution of worker.generate_stream(params), the create_background_tasks(request_id) function might not be called. This could lead to the semaphore not being properly released.
Related issue number (if applicable)
Closes #3389
Checks
- [x] I've run
format.shto lint the changes in this PR. - [x] I've included any doc changes needed.
- [x] I've made sure the relevant tests are passing (if applicable).
When you have a moment. Could you please take a look at this PR? @merrymercy @infwinston @BabyChouSr