flash-attention icon indicating copy to clipboard operation
flash-attention copied to clipboard

Published 20 hours ago • verified publisher icon Dao-AILab

[Rotary] more varlen rotary function implement

Open GGGGGGXY opened this issue 2 years ago • 10 comments

  1. add varlen supporting of MHA layer
  2. add varlen supporting of ApplyRotaryEmbQKV_ / ApplyRotaryEmbKV_ (with test code)
  3. fix some code format & spell error

GGGGGGXY avatar Sep 19 '23 17:09 GGGGGGXY

Thanks! Is the formatting by black using line length of 100?

tridao avatar Sep 20 '23 06:09 tridao

Thanks! Is the formatting by black using line length of 100?

sorry my bad. I am formatting by black with default setting. And now the format looks fine.

GGGGGGXY avatar Sep 20 '23 06:09 GGGGGGXY

Are there any plans to merge this soon? This feature would be really useful.

kklemon avatar Oct 07 '23 15:10 kklemon

@tridao Hi~ Sorry for disturbing. Is there still any problems? We looking forward to merge this pr to main~

GGGGGGXY avatar Oct 12 '23 02:10 GGGGGGXY

Sorry I've just been busy. Let me take a look tomorrow.

tridao avatar Oct 12 '23 02:10 tridao

@tridao Sorry for disturbing again. It would be greatly appreciated if this PR could be merged. Thank you for your time again.

GGGGGGXY avatar Oct 25 '23 07:10 GGGGGGXY

Hi, I'm curious about the status of this PR. This would be really useful to have merged!

scopello avatar Dec 26 '23 02:12 scopello

+1, would be really useful.

henrycharlesworth avatar Jan 31 '24 10:01 henrycharlesworth

@GGGGGGXY am I right in thinking this won't currently work for cross attention? How hard would it be to add support for this?

henrycharlesworth avatar Feb 08 '24 13:02 henrycharlesworth

Hi, any progress with this PR?

BakerBunker avatar Jul 17 '24 13:07 BakerBunker