Implement Flash Attention Option

[aeryncaen]

Would love to see a faster, more memory efficient attention implemented like Flash Attention. :)

[ggerganov]

In whisper.cpp I tried using FA in the Decoder and it did not help (it does help a lot in the Encoder). I guess it is a matter of the tensor sizes, but of course, maybe I didn't implement it properly.

https://github.com/ggerganov/whisper.cpp/pull/284

[Orevantum]

Is it possible to implement multi-query attention then?

In whisper.cpp I tried using FA in the Decoder and it did not help (it does help a lot in the Encoder). I guess it is a matter of the tensor sizes, but of course, maybe I didn't implement it properly.

ggerganov/whisper.cpp#284

[xloem]

also note in flexgen they use top 10% sparse attention

[jamesbiederbeck]

also note in flexgen they use top 10% sparse attention

Sparse attention is cool, but lossy. Flash attention is exact.