Tri Dao comments

Results 429 comments of


                                            Tri Dao

trafficstars

flash-attn NVIDIA CUDA `nvcc` error on HuggingFace spaces

You need nvcc.

dropout in FA3 needs get fixed?

FA3 doesn't have dropout

dropout in FA3 needs get fixed?

Not any time soon unless someone volunteers to work on it.

There was a strange computation error between standard attention and flash-attention2

The right thing to compare to is standard attention in fp32. In this case FlashAttention is actually **more** accurate than the standard implementation in fp16: ``` torch.manual_seed(0) batch_size = 1...

mamba2 error

Looks like a Triton error

HF version infer very slower than Original version?? How different between 2 version infer?

Idk how the HF version is implemented. We recommend the version in this repo.

FA-V3 deterministic usage

Oh it just hasn't been tested very well. dq semaphore should work except for hdim256. I'm not sure dk & dv (when GQA) semaphores have worked yet.

Fix +/-inf in LSE returned by forward

One issue I can see is that in the backward pass, if lse = +inf then exp(qk - lse) returns 0, which is what we want. If lse = -inf...

Applying FA3 in qwen2 model fine-tuning is slower than FA2

Are you using the latest commit? There's a recent update to enable causal for the backward. Can you profile to get the time for the attention kernel?

Applying FA3 in qwen2 model fine-tuning is slower than FA2

This commit: https://github.com/Dao-AILab/flash-attention/commit/bafe253042fb251a28f351ad0a2657da26263f31