vllm icon indicating copy to clipboard operation
vllm copied to clipboard

Published 20 hours ago • verified publisher icon vllm-project

Support for Falcon-7B / 40B models

Open sujithjoseph opened this issue 1 year ago • 5 comments

It would be great, if you can add support for Falcon models as well! Does it support onnx models today?

sujithjoseph avatar Jun 21 '23 17:06 sujithjoseph

Currently, vLLM does not support ONNX models. Supporting Falcon is on our roadmap. Thanks for your suggestion.

WoosukKwon avatar Jun 21 '23 19:06 WoosukKwon

@WoosukKwon When do you anticipate adding support for Falcon to vLLM?

MotzWanted avatar Jun 22 '23 08:06 MotzWanted

@MotzWanted I'm working on it now. I think we can add less-optimized version of Falcon (MQA replaced by MHA) quickly (within a few days) and then develop kernels to make the model actually use MQA.

WoosukKwon avatar Jun 24 '23 00:06 WoosukKwon

I'm waiting for supporting Falcon families, too. Thanks a lot for your works.

emphasis10 avatar Jun 24 '23 18:06 emphasis10

great, thanks

HSQ79815 avatar Jun 25 '23 08:06 HSQ79815

thanks, looking forward to it

teopapad92 avatar Jul 17 '23 10:07 teopapad92