bytedance
Question about W4A16 Benchmark
I benchmarked the downstream performance of the ABQ W4A16 (per-channel, symmetric) quantized version of the MiniCPM-1B-sft-llama-format model. I used a custom dialogue dataset for calibration and modified the input_ids and labels logic in datautils.py. However, the performance did not match that of AutoGPTQ (with act_order). Do you have any suggestions?
Hi, for the W4A16 configuration, most mainstream quantization algorithms can achieve good results. If using a per-channel setting for W4A16, you can choose any quantization algorithm you prefer. However, in more challenging settings like W3 or W2, AWQ/GPTQ will experience significant accuracy drops under per-channel settings, where the advantages of ABQ will become more apparent.
Hi, for the W4A16 configuration, most mainstream quantization algorithms can achieve good results. If using a per-channel setting for W4A16, you can choose any quantization algorithm you prefer. However, in more challenging settings like W3 or W2, AWQ/GPTQ will experience significant accuracy drops under per-channel settings, where the advantages of ABQ will become more apparent.
Hi, The w4a16 per-channel quantized models perform quite well on multiple choice benchmarks. However, when we look at benchmarks that require longer outputs, like gsm8k or humaneval, the performance doesn't seem to be as good. We're interested in pushing the limits of the w4a16 per-channel quantization methods. In this case, it's still a challenge because we've set higher expectations for its performance.