FastChat icon indicating copy to clipboard operation
FastChat copied to clipboard

Fix Semaphore release Issue in `api_generate_stream` Function of vllm_worker

Open coolbeevip opened this issue 1 year ago • 1 comments
trafficstars

Why are these changes needed?

This PR addresses a potential issue in the api_generate_stream function where the semaphore might not be properly released in all scenarios. Specifically, if an exception occurs during the execution of worker.generate_stream(params), the create_background_tasks(request_id) function might not be called. This could lead to the semaphore not being properly released.

Related issue number (if applicable)

Closes #3389

Checks

  • [x] I've run format.sh to lint the changes in this PR.
  • [x] I've included any doc changes needed.
  • [x] I've made sure the relevant tests are passing (if applicable).

coolbeevip avatar Jun 09 '24 03:06 coolbeevip

When you have a moment. Could you please take a look at this PR? @merrymercy @infwinston @BabyChouSr

coolbeevip avatar Jun 09 '24 03:06 coolbeevip