Why we need to calculate residual connections when visualize attention maps?

[JamenceTom]

Thanks for your great job! I am curious why we need to calculate residual connections when visualizing attention maps?

[vgoklani]

I'm curious too! Why do we need this?

[JimEverest]

Same question, hi @jeonsworld, could you please help to elaborate any specific reason for adding this identity matrix? Much appreciate.

[SeungHyun104]

In my opinion, In ViT's transformer module, It has residual connection. So from layer 1 to 12, Actual Attention map which ViT Model use is residual attention map.