localGPT icon indicating copy to clipboard operation
localGPT copied to clipboard

Published 20 hours ago • verified publisher icon PromtEngineer

How to load a quantized model ?

Open fmuntean opened this issue 2 years ago • 2 comments

How do I load the following model? TheBloke/vicuna-7B-1.1-GPTQ-4bit-128g

When I try I get he following error: TheBloke/vicuna-7B-1.1-GPTQ-4bit-128g does not appear to have a file named pytorch_model.bin, tf_model.h5, model.ckpt or flax_model.msgpack.

How Do I pass the correct parameters for the model to work?

fmuntean avatar Jun 05 '23 15:06 fmuntean

As far as I know, quantized models are not yet supported. I might be wrong.

PromtEngineer avatar Jun 06 '23 06:06 PromtEngineer

Yes, it looks like they are not supported, yet.

This is mentioned here: https://github.com/PromtEngineer/localGPT/issues/111#issuecomment-1583131451

alexfilothodoros avatar Jun 09 '23 09:06 alexfilothodoros