FastChat
FastChat copied to clipboard
Fix the "AttributeError: 'AsyncLLM' object has no attribute 'engine'"…
trafficstars
This makes vllm_worker work out of the box with vllm v0.8.3, which opens FastChat to work with all the recent models!
@merrymercy it's just a environment variable
Why are these changes needed?
VLLM changed since v.0.8.0, and there's a compatibility mode which is set with the VLLM_USE_V1 environment variable. If you disable it, legacy code runs again.
Related issue number (if applicable)
Closes #3704
Checks
- [x] I've run
format.shto lint the changes in this PR. - [x] I've included any doc changes needed.
- [x] I've made sure the relevant tests are passing (if applicable).