DJL 0.30 Sagemaker Endpoint Deployment using vllm of quantized model parameter option.quantization is not working

[adi7820]

Hello,

I'm trying to deploy LLAMA 3.2 vision 4 bit bitsandbytes quantized model as a sagemaker endpoint, but I've encountered one error regarding quantization.

As per the above image it says it is receiving quantization as 'None' even though I've set it's properties during configuration while creating serving.properties of sagemaker endpoint.

%%writefile serving.properties engine=Python option.model_id=unsloth/Llama-3.2-11B-Vision-Instruct-bnb-4bit option.rolling_batch=vllm option.dtype=bf16 option.max_model_len=8192 option.max_num_seqs=1 option.enforce_eager=True option.gpu_memory_utilization=0.9 option.quantization=bitsandbytes option.load_format=bitsandbytes

When I re-iterated the error in vllm github repo then I found that the actual reason of the error that parameter quantization is not receiving its value.

Can you guys help me with the possible solution or need to wait for another version?