FastChat icon indicating copy to clipboard operation
FastChat copied to clipboard

update vllm_worker.py

Open jbding opened this issue 1 year ago • 0 comments

Why are these changes needed?

  • The vllm_worker code does not catch exceptions: when I check vllm's SamplingParams function, there is parameter verification and the error ValueError is thrown out.
  • Because there is no capture, the semaphore is not released after the code execution fails, causing the model to fail to work.

Related issue number (if applicable)

Fix #3328

Checks

  • [ ✓] I've run format.sh to lint the changes in this PR.
  • [ ✗] I've included any doc changes needed.
  • [ ✓] I've made sure the relevant tests are passing (if applicable).

jbding avatar May 13 '24 08:05 jbding