Attention maps exhibit outliers even with registers

[XiphosF]

Hi, as discussed in #306, the large model consistently shows worse attention maps than the base and medium models. Adding registers removes attention outliers on some images, but others still have them. Is this normal behavior, and has anyone else experienced it with the large model with registers?

[Ahmaddimran]

@XiphosF got any answer for it?

[XiphosF]

One thing that enhanced my attention maps was to check and reinstall the xformers library with the correct version. Didn't fix everything, but still was notably better.