LLaVA-NeXT icon indicating copy to clipboard operation
LLaVA-NeXT copied to clipboard

Published 20 hours ago • verified publisher icon LLaVA-VL

How to deploy this model via API?

Open Iven2132 opened this issue 1 year ago • 2 comments

How do we deploy this model via API? Can I deploy it on vLLM or lmdeploy? I can't find any example to run this with HuggingFace transformers.

I want to deploy 72b and 110b model

Iven2132 avatar May 11 '24 07:05 Iven2132

Hi @Iven2132 , see that you have already noticed our PR in sglang.

For others that have similar problems and reading this issue, you can refer to here

kcz358 avatar May 11 '24 09:05 kcz358

Would be ideal to have TGI and vLLM support as well.

I tried TGI but it seems that won't work as the model isn't recognised.

RonanKMcGovern avatar May 16 '24 10:05 RonanKMcGovern