Ivan Zhang

Accuracy drop after `fuse_layer_norms`

1

I encountered an unexpected precision loss while using Quarot. I conducted comparison experiments on LLaMA-2-7b: Performing w4a16 RTN quantization on the model resulted in a PPL (Perplexity) of 7.354664. Performing...

Precision Loss in bf16 Model with float32 Rotation Calculation

2

Description: I am experiencing a significant precision drop when using the quarot algorithm on a device limited to float32 calculations. Originally designed for double precision, the rotations are cast to...