Add FlashAttention v3 support

Open rasbt opened this issue 1 year ago • 2 comments

This adds FlashAttention v3 support for H100 and H800 GPUs.

We should wait until they release the full code with bf16 supported and it is tested on an H100 machine.

Jul 12 '24 19:07 rasbt

We should wait until they release the full code with bf16 supported and it is tested on an H100 machine.

Has it been done? Maybe it's better to keep it in a Draft mode till all the testing is completed?

Jul 17 '24 12:07 Andrei-Aksionov

Maybe it's better to keep it in a Draft mode till all the testing is completed?

Sure, I can move that to draft mode. But like I said in the original text,

We should wait until they release the full code with bf16 supported and it is tested on an H100 machine.

Jul 17 '24 12:07 rasbt