metaseq icon indicating copy to clipboard operation
metaseq copied to clipboard

Integrate LucidRain's RotaryEmbeddings

Open suchenzang opened this issue 1 year ago • 2 comments

See https://github.com/lucidrains/rotary-embedding-torch/blob/main/rotary_embedding_torch/rotary_embedding_torch.py

And from PaLM paper:

We use RoPE embeddings (Su et al., 2021) rather than absolute or relative position embeddings, since RoPE embeddings have been shown to have better performance on long sequence lengths.

suchenzang avatar Jan 27 '23 07:01 suchenzang