vllm
vllm copied to clipboard
vllm-project
PowerInfer : using a combination of cpu and gpu for faster Inference
Splitting hot and cold neurons across cpu and gpu allows faster Inference when using larger models/higher quantisations. Demo shows 11x speedup over llama.cpp when using a 40b on a single 24gb GPU.
Demo https://twitter.com/omarsar0/status/1737168751668187229?t=blU8xZMb7JMJTtAHra7zvQ&s=19
GitHub https://github.com/SJTU-IPADS/PowerInfer
Wondering if this is something that can also be integrated into vllm.
It is designed to improve speed of mainly sparse LLMs. It won't allow faster inference with dense LLMs.
They show sparsity even in dense models like falcon. But I guess mixtral MoE is a better candidate
It is designed to improve speed of mainly sparse LLMs. It won't allow faster inference with dense LLMs.
but there still many LLMs using the ReLU activation function, so this can still have a chance?
