Tri Dao
Tri Dao
`max_seqlen_*` is no longer necessary in FA4. Are you running the fwd or fwd & bwd?
can you create a PR?
FA3 is a beta release, you'd need to call a different interface to use it (for now). https://github.com/Dao-AILab/flash-attention/blob/main/hopper/flash_attn_interface.py As we roll it out (e.g. after v3.0.0 tag) eventually the old...
FA3 now doesn't require tensors to be contiguous.
Please make sure you use the latest commit (e.g. uninstall previous versions). The newest code doesn't do any contiguous check anymore.
Sorry I mispoke, let me update to remove those checks.
done, thanks for checking
`qkvpacked` avoids the concat in the backward pass (backward of unbind is concat).
As it says, contiguous last dimension. I.e. last stride should be 1.