AutoGPTQ icon indicating copy to clipboard operation
AutoGPTQ copied to clipboard
verified publisher icon AutoGPTQ

[BUG]Unable to quantize Falcon-7b

Open wchen61 opened this issue 1 year ago • 3 comments

Describe the bug

https://huggingface.co/tiiuae/falcon-7b

Unable to quantize Falcon-7b model, throws an assertion error. auto_gptq/nn_modules/qlinear/qlinear_exllama.py: line 69 assert infeatures % self.group_size == 0

auto_gptq v0.7.1 transformers 4.40.0

wchen61 avatar Aug 27 '24 02:08 wchen61