Long-Range-Grouping-Transformer icon indicating copy to clipboard operation
Long-Range-Grouping-Transformer copied to clipboard

LGA grouping strategy

Open Tcode8 opened this issue 6 months ago • 4 comments

Thank you for your excellent work. While studying your work, I was not very familiar with the implementation of LGA grouping code. Could you please explain to me what these dimensional changes represent and how they reflect the grouping of different blocks ‘’‘ q_cls, k_cls, v_cls = map(lambda t: rearrange(t, '(b v) h (t1 s1 t2 s2) d -> b (s1 s2) h (t1 t2 v) d', b=b_s, v=view_num, t1=token_num, s1=self.group_num, t2=token_num, s2=self.group_num), (q, k, v)) ’‘’ looking for your reply!

Tcode8 avatar Aug 10 '24 09:08 Tcode8