FlyPanda

How to use Medusa to support non llama models?

8

### System Info Hardware: L20 Version: 0.11.0.dev20240625 Model: Bloom7b1 ### Who can help? @ncomly-nvidia @byshiue I have obtained the Medusa head for Bloom according to the official Medusa documentation, but...

### System Info GPU：L20 Tensorrt-LLM：v0.11.0 transformers: 4.42.0 ### Who can help? @ncomly-nvidia @kaiyux prompt='你好，请介绍一下喜马拉雅山的详细信息' ### 1、transformers about params: generation_config = GenerationConfig( top_k=1, temperature=1, max_length=2048, max_new_tokens=80, repetition_penalty=1.0, early_stopping=True, do_sample=True, num_beams=1, top_p=1,...