Have you tried visualizing the attention maps?

[tegillis]

In my experiments the attention maps are not showing anything meaningful. I'm also not sure how you would do it with the attention module in the middle.

[taki0112]

The author code is not public, so I just decided where to position it.

[karlhugle]

The self-attention is based on "The image features from the previous hidden layer." at least it is right to place after the first conv layer.