Does FlashAttention 2 use memory coalescing in Nvidia GPU?

[AndroidSheepy]

trafficstars

Dear developers of flash-attention, thank you for your great work.

Recently I used Nvidia Nsight Compute to profile the performence of FlashAttention 2 kernel(package version is v2.4.2, hardware is A100), and in the memory performance section of the generated report it says the memory access pattern is not optimal because of the uncoalesced memory read/load in FlashAttention kernel(You can check the sentences in the red box in the following photo ):

Based on the results in the photo, I'm curious that does flash-attention 2 notice to use memory coalescing when accessing the GPU memory? It would be very helpful if you reply:).

[tridao]

Most of the reads/writes are coalesced. There are some small writes (e.g. writing to the LSE) that are not, but I don't think it matters. Lmk if you profile more and have more info on how much some of these un-coalesced access makes a difference.