Hello! May I inquire if the LayerNorm module has not been quantized in this model?

Open GoatWu opened this issue 2 years ago • 3 comments

Aug 15 '23 09:08 GoatWu

Hi,

In this work, the output (activation) of LayerNorm is quantized, while retaining the floating-point computation of LayerNorm itself.

Aug 15 '23 11:08 zkkli

Thank you! And I have another question. In FQ-ViT, both the softmax and LayerNorm layers are computed in integer form. So, would it be unfair to compare the accuracy of the method in this paper with FQ-ViT?

Aug 15 '23 13:08 GoatWu

Our method follows the settings of the previous PTQ4ViT and APQ-ViT, so comparisons with the PTQ4ViT and APQ-ViT are exactly fair.

Aug 16 '23 06:08 zkkli