FastChat icon indicating copy to clipboard operation
FastChat copied to clipboard

Fix the "AttributeError: 'AsyncLLM' object has no attribute 'engine'"…

Open surak opened this issue 7 months ago • 0 comments
trafficstars

This makes vllm_worker work out of the box with vllm v0.8.3, which opens FastChat to work with all the recent models!

@merrymercy it's just a environment variable

Why are these changes needed?

VLLM changed since v.0.8.0, and there's a compatibility mode which is set with the VLLM_USE_V1 environment variable. If you disable it, legacy code runs again.

Related issue number (if applicable)

Closes #3704

Checks

  • [x] I've run format.sh to lint the changes in this PR.
  • [x] I've included any doc changes needed.
  • [x] I've made sure the relevant tests are passing (if applicable).

surak avatar Apr 08 '25 17:04 surak