swift icon indicating copy to clipboard operation
swift copied to clipboard

多模态模型(qwen_vl_chat)量化失败

Open Luccadoremi opened this issue 1 month ago • 0 comments

使用以下命令对qwen_vl_chat多模态模型量化失败,请问目前支持多模态模型的量化吗? ############################## CUDA_VISIBLE_DEVICES=6 swift export
--model_type qwen-vl-chat
--quant_bits 4
--dataset coco-en-mini
--quant_method gptq
--quant_output_dir output/quant #############################

报错信息 ############################ Traceback (most recent call last): File "/miniconda/lib/python3.9/site-packages/swift/cli/export.py", line 5, in export_main() File "/miniconda/lib/python3.9/site-packages/swift/utils/run_utils.py", line 27, in x_main result = llm_x(args, **kwargs) File "/miniconda/lib/python3.9/site-packages/swift/llm/export.py", line 113, in llm_export gptq_quantizer = gptq_model_quantize(model, template.tokenizer) File "/miniconda/lib/python3.9/site-packages/swift/llm/export.py", line 87, in gptq_model_quantize gptq_quantizer.quantize_model(model, tokenizer) File "/miniconda/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/miniconda/lib/python3.9/site-packages/optimum/gptq/quantizer.py", line 436, in quantize_model model(**data) File "/miniconda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/miniconda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(*args, **kwargs) File "/root/.cache/huggingface/modules/transformers_modules/Qwen-VL-Chat/modeling_qwen.py", line 856, in forward transformer_outputs = self.transformer( File "/miniconda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/miniconda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl return forward_call(*args, **kwargs) File "/root/.cache/huggingface/modules/transformers_modules/Qwen-VL-Chat/modeling_qwen.py", line 557, in forward assert (bos_pos[0] == eos_pos[0]).all() RuntimeError: The size of tensor a (7) must match the size of tensor b (6) at non-singleton dimension 0

Luccadoremi avatar May 28 '24 10:05 Luccadoremi