long-context topic

Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch

lucidrains

Implementation of Recurrent Memory Transformer, Neurips 2022 paper, in Pytorch

lucidrains

RAN: Recurrent Attention Networks for Long-text Modeling | Findings of ACL23

4AI

[ACL 2024] LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding

THUDM

Codes for the paper "∞Bench: Extending Long Context Evaluation Beyond 100K Tokens": https://arxiv.org/abs/2402.13718

OpenBMB

Implementation of paper "LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models"

Glaciohound

Implementation of Perceiver AR, Deepmind's new long-context attention network based on Perceiver architecture, in Pytorch

lucidrains

Transformers with Arbitrarily Large Context

forhaoliu

My own attempt at a long context genomics model, leveraging recent advances in long context attention modeling (Flash Attention + other hierarchical methods)

lucidrains

LongQLoRA: Extent Context Length of LLMs Efficiently

yangjianxin1