vllm icon indicating copy to clipboard operation
vllm copied to clipboard

Published 20 hours ago • verified publisher icon vllm-project

CTranslate2

Open Matthieu-Tinycoaching opened this issue 1 year ago • 2 comments

Hello,

Thanks for the great framework for deploying LLM.

Would it be possible to use a LLM model compiled with the CTranslate2 library?

Matthieu-Tinycoaching avatar Jun 22 '23 16:06 Matthieu-Tinycoaching

Thanks for bringing this up. We will investigate the CTranslate2 library and evaluate the difficulty and the potential benefit of adding it into vLLM.

zhuohan123 avatar Jun 23 '23 08:06 zhuohan123

Would love to see this, ct2 would be a great integration! It would give us easy access to fast 8 bit inference and plays nice with HF Transformers. Thank you for the library so far!!

anujnayyar1 avatar Jun 24 '23 03:06 anujnayyar1

Hi,

Any news regarding this integration? Ctranslate2 has already proven its speed within the TitanML framework for local LLM serving.

Matthieu-Tinycoaching avatar Aug 02 '23 08:08 Matthieu-Tinycoaching

hi,

any news on this?

manishiitg avatar Oct 02 '23 08:10 manishiitg

+1

Matthieu-Tinycoaching avatar Oct 08 '23 09:10 Matthieu-Tinycoaching

+11

shixianc avatar Oct 21 '23 00:10 shixianc

@zhuohan123 do you see any benefit of adding this to vLLM?

hmellor avatar May 18 '24 10:05 hmellor