transformers icon indicating copy to clipboard operation
transformers copied to clipboard

Published 20 hours ago • verified publisher icon huggingface

RingAttention Support

Open Hambaobao opened this issue 1 year ago • 0 comments

Feature request

Hello,

I would like to inquire about the potential inclusion of RingAttention in Transformers, which could enable training with longer sequences.

Motivation

The incorporation of RingAttention would significantly enhance the capabilities for users engaged in advanced projects involving LLMs and LMMs. I believe this feature would make Transformers even more versatile and valuable for the research and development community.

Your contribution

References:

Hambaobao avatar Feb 17 '24 06:02 Hambaobao