NAFNet icon indicating copy to clipboard operation
NAFNet copied to clipboard

Published 20 hours ago • verified publisher icon megvii-research

LayerNorm's position in NAFBlock

Open four1000 opened this issue 3 years ago • 1 comments

Hi,

refer from "Attention Is All You Need", the LN position is in the tail of the attention and feedforward block, but you put the LN at the head of attention and feedforward in the case of NAFBlock. Is there any theory or designing idea or just by the experiment result?

four1000 avatar May 06 '22 08:05 four1000

Hi, four1000,

Thanks for your attention to NAFNet. We put LN at the head of "attention" / ffn following Swin Transformer, Restormer, and, etc. It works fine, so we did not ablate the position of LN.

mayorx avatar May 06 '22 15:05 mayorx