Flash Attention V1 support

Open Murad-Awad opened this issue 1 year ago • 4 comments

I noticed this repo: https://github.com/huggingface/candle-flash-attn-v1. Was curious if there is any plan on the roadmap to have a feature allowing flash-attn-v1 (rather than v2) in order to support a wider range of gpus.

Jan 18 '25 01:01 Murad-Awad

Maybe @LaurentMazare ?

Feb 12 '25 16:02 Murad-Awad

Bump on this if possible.

Apr 02 '25 17:04 Murad-Awad

@LaurentMazare @EricLBuehler can you please advise per this?

May 07 '25 20:05 Murad-Awad

Hey @Murad-Awad! We have candle-extensions now, and you can use the candle-flash-attn-v1 crate. The function is a 1:1 drop-in replacement for the v2 implementation here in Candle.

Let me know if you have any issues using this.

May 08 '25 01:05 EricLBuehler