ring-flash-attention icon indicating copy to clipboard operation
ring-flash-attention copied to clipboard

Published 20 hours ago • verified publisher icon zhuzilin

Does ring-attn not support dropout?

Open chinapanda opened this issue 9 months ago • 3 comments

In the backward function of ring-attn, rng_state does not use the value from forward function, but directly passes in None. Does this indicate that ring-attn does not support dropout?

chinapanda avatar May 20 '24 10:05 chinapanda