vllm icon indicating copy to clipboard operation
vllm copied to clipboard

Published 20 hours ago • verified publisher icon vllm-project

Support for fastchat-t5-3b-v1.0

Open Matthieu-Tinycoaching opened this issue 2 years ago • 1 comments

It would be great if you could support fastchat-t5-3b-v1.0, which is a derivation of Flan-T5-XL model: https://huggingface.co/lmsys/fastchat-t5-3b-v1.0

Matthieu-Tinycoaching avatar Jun 23 '23 11:06 Matthieu-Tinycoaching

Hi @Matthieu-Tinycoaching, thanks for bringing it up! As mentioned in #187, T5 support is definitely on our roadmap. The current blocker is its encoder-decoder architecture, which vLLM's current implementation does not support. As it requires non-trivial modifications to our system, we are currently thinking of a good design to support it in vLLM.

WoosukKwon avatar Jun 24 '23 00:06 WoosukKwon

Closing as a duplicate of #187

hmellor avatar Mar 06 '24 10:03 hmellor