unsloth with vllm in 8/4 bits

[quancore]

I have trained qlora model with unsloth and I want to serve with vllm but I did not found a way to serve model in8/4 bits ?

[danielhanchen]

@quancore I'm not sure / unsure if vLLM allows serving in 4 or 8 bits! 16bit yes, but unsure on 4 or 8

[quancore]

@danielhanchen I think it is: https://github.com/vllm-project/vllm/issues/1155

[patleeman]

@danielhanchen I think it is: vllm-project/vllm#1155

Looks like they only support AWQ quantization not via bitsandbytes.

[danielhanchen]

@patleeman Oh ye AWQ is great - I'm assuming you want to quantize it to AWQ?

[quancore]

@patleeman @danielhanchen well yes, maybe we should support AWQ so we can use qlora models with vllm?

[marcelodiaz558]

Hello there. I am also interested in using with VLLM a 8/4 bits model trained with Unsloth. Currently, it works fine with 16 bits but requires too much VRAM. Is there a way to quantize a model trained with Unsloth using AWQ or GPTQ?