flash-attention icon indicating copy to clipboard operation
flash-attention copied to clipboard

Published 20 hours ago • verified publisher icon Dao-AILab

dropout in FA3 needs get fixed?

Open jundaf2 opened this issue 1 year ago • 1 comments
trafficstars

it seems the current flash-attn-3 gives different dropout mask in its forward and backward kernel.

jundaf2 avatar Sep 30 '24 09:09 jundaf2

FA3 doesn't have dropout

tridao avatar Sep 30 '24 09:09 tridao

i see, thanks. will FA3 officially support dropout?

jundaf2 avatar Oct 09 '24 04:10 jundaf2

Not any time soon unless someone volunteers to work on it.

tridao avatar Oct 09 '24 06:10 tridao

i see, thanks!

jundaf2 avatar Oct 10 '24 09:10 jundaf2