asappresearch
Use Rotary Embeddings in SRU++
@taolei87 I'm way too excited with SRUpp. I already adopted in all my projects with impressive results.
Recently the great people at Eleuther.ai found a silver-bullet in transformers. The Rotary Embeddings.
The RoPE is a relative position encoding method with promise theoretical properties. The main idea is to multiply the context embeddings (q,k in the Transformer) by rotation matrices depending on the absolute position. One can prove that the inner product of the context embeddings will become only depending on the relative position.
It gave a performance increase in virtually all tasks tested. There is more information here: https://blog.eleuther.ai/rotary-embeddings/
From the little of I understood of SRU++ builds in self-attention, it would be possible to use in SRUppAttention to q, k. Do you think that worth the test?
Thank you @taoleicn . I did a very possibly wrong implementation here https://github.com/bratao/sru/commit/1c614c34713a699451c60986afa2d9b0d3d86cba However, I'm running some tests and apparently it is converging faster
Thanks for sharing @bratao. I read the paper and it is indeed very interesting. Do you have any preliminary results? If you see consistent or significant improvement using RoE, we should find a way to incorporate it as an optional module.
@taoleicn I only tested on toy examples and them converged faster. Unfortunately, I'm too overwhelmed with work to test on something like enwik8
Did you had the opportunity to check if I placed on the right place on my implementation? I have a big impression that I placed on the wrong place.