Medusa vLLM support

Sep 20 '23 12:09 MichaelJayW

Are there any updates on this one?

Oct 11 '23 05:10 Data-drone

+1

Oct 16 '23 12:10 louisoutin

+1

Oct 23 '23 07:10 ruidongtd

+1

Nov 03 '23 03:11 insist93

+1

Nov 30 '23 02:11 leonardxie

+1

Dec 18 '23 14:12 TexasRangers86

So, could you provide advice so that I can make custom modifications on vLLM myself (llama2 70b)?

Dec 20 '23 01:12 Lvjinhong

fwiw, i know this is about vLLM, but you can run medusa on tgi using --speculate 3

Dec 21 '23 11:12 RonanKMcGovern

fwiw, i know this is about vLLM, but you can run medusa on tgi using --speculate 3

hello，how can I pass medusa model and base model args when I use medusa on tgi.

Dec 26 '23 12:12 TexasRangers86

fwiw, i know this is about vLLM, but you can run medusa on tgi using --speculate 3

hello，how can I pass medusa model and base model args when I use medusa on tgi.

Just pass the medusa model repo (as you would with any other model) and then add on --speculate 2

You can try this template: https://runpod.io/gsc?template=2xpg09eenv&ref=jmfkcdio

Dec 26 '23 12:12 RonanKMcGovern

Thanks a lot !!!!

Dec 26 '23 12:12 TexasRangers86

how to use medusa based on vllm or sglang?

Sep 29 '24 08:09 chuangzhidan