rotary-embedding-torch icon indicating copy to clipboard operation
rotary-embedding-torch copied to clipboard

Published 20 hours ago • verified publisher icon lucidrains

RoPE-Mixed: Improvement over Axial for n-D

Open tasansal opened this issue 1 year ago • 1 comments

Hi @lucidrains,

These folks talk about improving axial-RoPE performance. Some comparisons to axial-RoPE look nice, but for some, I am not convinced. I wanted to get your thoughts on this. If it makes sense, can we integrate this into the repo?

https://arxiv.org/abs/2403.13298

tasansal avatar Jun 27 '24 15:06 tasansal