unsloth
unsloth copied to clipboard
unsloth with vllm in 8/4 bits
I have trained qlora model with unsloth and I want to serve with vllm but I did not found a way to serve model in8/4 bits ?
@quancore I'm not sure / unsure if vLLM allows serving in 4 or 8 bits! 16bit yes, but unsure on 4 or 8
@danielhanchen I think it is: https://github.com/vllm-project/vllm/issues/1155
@danielhanchen I think it is: vllm-project/vllm#1155
Looks like they only support AWQ quantization not via bitsandbytes.
@patleeman Oh ye AWQ is great - I'm assuming you want to quantize it to AWQ?
@patleeman @danielhanchen well yes, maybe we should support AWQ so we can use qlora models with vllm?
Hello there. I am also interested in using with VLLM a 8/4 bits model trained with Unsloth. Currently, it works fine with 16 bits but requires too much VRAM. Is there a way to quantize a model trained with Unsloth using AWQ or GPTQ?