optimum export openvino ,use gptq&scale_estimation error

[nnbw-liu]

Exception has occurred: ValueError could not broadcast input array from shape (84934656,) into shape (9216,) File "C:\Users\admin\Desktop\convert_model.py", line 15, in model = OVModelForCausalLM.from_pretrained(model_id, export=True,quantization_config=quantization_config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ValueError: could not broadcast input array from shape (84934656,) into shape (9216,)

source code:

from optimum.intel import OVModelForCausalLM,OVWeightQuantizationConfig from transformers import AutoTokenizer

model_id = "./Phi-3.5-mini-instruct" quantization_config = OVWeightQuantizationConfig( bits=4, sym=True, quant_method="awq", scale_estimation=True, group_size=-1, gptq=True, dataset="wikitext2" )

model = OVModelForCausalLM.from_pretrained(model_id, export=True,quantization_config=quantization_config) tokenizer = AutoTokenizer.from_pretrained(model_id) save_directory = "./Phi3.5-ov-awq-gptq" model.save_pretrained(save_directory) tokenizer.save_pretrained(save_directory)

What shall I do?

[nnbw-liu]

optimum-cli export openvino --model .\Phi-3.5-mini-instruct\ --task text-generation-with-past --weight-format int4 --group-size -1 --sym --awq --dataset wikitext2 --gptq --scale-estimation phi-3.5-mini-int4-awq-gptq-scale

It has the same effect.

[nikita-savelyevv]

Hi @qihui-liu . I apologize for the late response. The command below exports the model correctly on my side. Could you please try it?

optimum-cli export openvino --model microsoft/Phi-3.5-mini-instruct --task text-generation-with-past \
    --weight-format int4 --group-size -1 --sym --awq --dataset wikitext2 --gptq --scale-estimation \
    phi-3.5-mini-int4-awq-gptq-scale

[nnbw-liu]

Thank you for your reply. I have completed the conversion of the model.