Mega-pytorch
Mega-pytorch copied to clipboard
lucidrains
Implementation of Mega, the Single-head Attention with Multi-headed EMA architecture that currently holds SOTA on Long Range Arena
1. For https://arxiv.org/pdf/2209.10655.pdf#page=21 , why use `x = sqrt(2)` specifically ? why is it not easier to just use `x = 1` ?  2. In https://arxiv.org/pdf/2109.08668.pdf#page=5 , I do...
Metadata
Owner
Metadata
Implementation of Mega, the Single-head Attention with Multi-headed EMA architecture that currently holds SOTA on Long Range Arena