snowfall icon indicating copy to clipboard operation
snowfall copied to clipboard

Rotary positional embeddings for conformer

Open pzelasko opened this issue 3 years ago • 0 comments

Just throwing the idea around in case somebody wants to pick it up, I wanted to do it for some time but can't find the time. There's a new position embedding type called RoPE that yield improvements across different NLP tasks and now also ASR. Shouldn't be too difficult to add, and there is an implementation here: https://github.com/lucidrains/rotary-embedding-torch

Original paper: https://arxiv.org/abs/2104.09864 Application in ASR: https://arxiv.org/abs/2107.05907

pzelasko avatar Jul 21 '21 15:07 pzelasko