[Feature]: BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences

[sorasoras]

🚀 The feature, motivation and pitch

I through this is gonna be useful for serving larger scale of user so it should be useful for this project It should be quite useful for inference when scale up.

Alternatives

https://arxiv.org/abs/2403.09347

Additional context

No response

[AlpinDale]

I doubt this applies to inference as much as it does for training. Admittedly, I haven't given the paper a thorough read yet.