Filip Szatkowski

LwM - no gradient in attention distillation loss

5

Hi, when experimenting with LwM in FACIL I noticed that the method behaves the same regardless of the choice of `gamma` parameter that controls attention distillation loss. Upon closer investigation,...