vllm
vllm copied to clipboard
vllm-project
Feature request：support ExLlama
ExLlama (https://github.com/turboderp/exllama)
It's currently the fastest and most memory-efficient executor of models that I'm aware of.
Is there an interest from the maintainers in adding this support?
How do you plan on adding batched support for Exllama? I am very interested in your approach as I am trying to work on that too
ExLlamaV2 has taken over ExLlama in quantization performance for most cases. I hope we can get it implemented in vLLM because it is also an incredible quantization technique. Benchmarks between all the big quantization techniques indicate ExLlamaV2 is the best out of all of them. Have there been any new developments since it was added to the roadmap?
Please, having exllamav2 with paged attention and with continuous batching would be a big win for the LLM world
I was hoping this would be possible, too. I recently worked with the Mixtral-8x7b Model; AWQ 4-bit had significant OOM / Memory overhead compared to ExLlama2 in 4-Bit; also I ended up just running the model in 8-bit using ExLlama2, since that turned out to be the best compromise between model capabilities and VRAM footprint. I can run it in 8-bit on 3x3090 and use full 32k context with ExLlama2; but I need 4x3090 to be even able to load it in 16-bit within VLLM; and i reach OOM when I try to use full context.
So this would definitely be an amazing addition to have more flexibility in terms of VRAM-Resources.
