TWO BUGS in function get_qparams() in quant.py

[sasha-hailo]

Dear LLMC team, Thank you for your continuous effort to maintain this useful repo. I've been using it for a long period, and would like to draw your attention to two issues in the core of its quantization logic.

Current implementation of get_qparams() of class BaseQuantizer:

def get_qparams(self, tensor_range, device):
	min_val, max_val = tensor_range[0], tensor_range[1]
	qmin = self.qmin.to(device)
	qmax = self.qmax.to(device)
	if self.sym:
		abs_max = torch.max(max_val.abs(), min_val.abs())
		abs_max = abs_max.clamp(min=1e-5)
		scales = abs_max / qmax
		zeros = torch.tensor(0.0)
	else:
		scales = (max_val - min_val).clamp(min=1e-5) / (qmax - qmin)
		zeros = (qmin - torch.round(min_val / scales)).clamp(qmin, qmax)
		if not self.round_zp:
			zeros = qmin - (min_val / scales)
	return scales, zeros, qmax, qmin

Problem 1: Taking into account the asymmetry of symmetric representation In symmetric integer quantization using b bits, the actual range is between qmin = -2**(b-1) and qmax = 2**(b-1) -1. For instance, for b=4, we have qmin=-8 and qmax=7. I encountered the when trying to "requantize" weights that were already quantized to symmetric W4. Surprisingly, the "requantized" weights differed from the inputs, and reached worse accuracy. Reason: if the full range of the weights is used, we have min_val=-8s and max_val=7s (for some scalar s). Therefore, abs_max = 8s, and we have scales = abs_max / qmax = 8/7*s. As results, the "requantized" weights will only occupy 15 values between -7 to +7, and the value of qmin=-8 is never used.

My proposed change (in symmetric quantization case):

if self.sym:
	max_neg = torch.clamp(min_val, max=0.0)
	max_pos = torch.clamp(max_val, min=0.0)
	pos_scales = max_pos / qmax
	neg_scales = max_neg / qmin
	scales = torch.max(pos_scales, neg_scales)
	zeros = torch.tensor(0.0)

With the proposed correction, "requantization" of previously quantized weights won't cause any change in weights or accuracy.

Problem 2: Proper asymmetric quantization of dynamic ranges that do not include zero Note: I assume that you require the same uint representation for zero-point as for the target quantized values, since your code explicitly enforces: zeros = (qmin - torch.round(min_val / scales)).clamp(qmin, qmax) Since dequantization is given by x_dequant = scales * (x_quant - zeros), the assumption above means that the zero (0.0) value MUST reside inside the [de]quantized dynamic range. Therefore, the range between min_val and max_val must contain zero, otherwise the functionality above is incorrect. (Try to think what happens if min_val > 0: zeros will be clamped to zero, and as result, severe errors will be caused)

My proposed modification to the code in asymmetric quantization case (added two lines):

else:
	min_val = torch.min(min_val, torch.tensor(0.0).to(device))
	max_val = torch.max(max_val, torch.tensor(0.0).to(device))
	scales = ...
	zeros = ...

What do you think?