efficient-attention topic

Unofficial PyTorch implementation of the paper "cosFormer: Rethinking Softmax In Attention".

davidsvy

Pytorch implementation of "Compact Global Descriptor for Neural Networks" (CGD).

HolmesShuan

Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch

lucidrains

Efficient Infinite Context Transformers with Infini-attention Pytorch Implementation + QwenMoE Implementation + Training Script + 1M context keypass retrieval

jlamprou

Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch

lucidrains