Tri Dao comments

Results 490 comments of


                                            Tri Dao

torch.version = 2.2.0+cu118 but showing Error: FlashAttention is only supported on CUDA 11.6 and above.

Does `nvcc -V` run?

torch.version = 2.2.0+cu118 but showing Error: FlashAttention is only supported on CUDA 11.6 and above.

As mentioned in the README, we require CUDA (and nvcc) >= 11.6.

Implemented Flash Attention2 for Intel GPU hardware

Thanks for this contribution! What happens if user calls a function that's not currently supported (e.g. paged KV or varlen)?

There is no cu123 but cu124 for PyTorch

No mistake

There is no cu123 but cu124 for PyTorch

12.3 will work

There is no cu123 but cu124 for PyTorch

cuda minor version are compatible

value instability when training the model with AMD ROCm GPU

@rocking5566 does the AMD version support alibi?

Apple Silicon Support

I personally have no bandwidth for that, so we'd need folks to contribute.

Why do the output results of flash attention and muti head attention differ significantly under the same parameters

What's the difference? The right comparison is (flashattn in fp16 - reference implementation in fp32) vs (rerefnece implementation in fp16 - reference in fp32)

Make alibi slopes a trainable parameter?

No that's not implemented (one would have to change the backward pass code to compute the gradient of the slopes).