ValueError: Unknown quantization method: bitsandbytes. Must be one of ['awq', 'gptq', 'squeezellm', 'marlin'].

[manliu1225]

When I want to inference the finetuned model with vLLM, I got this error. I have saved unsloth finetuned model to HF model already. vLLM==0.4.0+cu118 unsloth==2024.5 transformers==4.40.2

[danielhanchen]

Oh you cannot use 4bit models - you must use model.save_pretrained_merged to 16bit then use vLLM