After quantization，the ppl is ok but humaneval score drops sharply

[ehuaa]

After using AutoAWQ quantizing my finetuned version model of qwen1.5-72b, i make two tests.

1. run ppl after quant for test 1
2. human eval test for test 2

for test 1, the ppl are almost the same, but the result of human eval test, the quantized model drops sharply which from 0.67 to 0.37, can you give some advice how to debug with this situation?

[casper-hansen]

You will probably have to quantize your model using a custom dataset for coding.

[WanBenLe]

In sometimes will rasie error in the quantization stage (all losses are nan), increase block_size (like 1024) in calib_data.py can improve the performance,

https://github.com/casper-hansen/AutoAWQ/blob/5f3785dcaa107ca76f5fa5355f459370c86f82d6/awq/utils/calib_data.py https://github.com/casper-hansen/AutoAWQ/issues/376