FlagAttention icon indicating copy to clipboard operation
FlagAttention copied to clipboard

Published 20 hours ago • verified publisher icon FlagOpen

Is it possible for your team to implement xformers.ops.memory_efficient_attention?

Open radna0 opened this issue 1 year ago • 5 comments

radna0 avatar Jul 11 '24 06:07 radna0

Thank you. I think the main difference between our implementation of flash_attn is that it takes an extra input, the attention bias. We can take a while to add this feature.

iclementine avatar Jul 15 '24 07:07 iclementine

Thank you @iclementine! Will it take a long time for you to implement this? I'm trying to run this on AMD GPUs and have had some success with HIP backend of Triton. Do you think it's possible to run on both Nvidia and AMD? What about performance difference?

radna0 avatar Jul 15 '24 07:07 radna0

I don't have a AMD GPU. Maybe there are some issues to run it on triton with other backends(some configs exceeding resource limits, some passed are not supported, etc). If you have some modifications to make it run on AMD GPUS, please inform us. Thank you.

I would take about 1~2 weeks to implement this, regarding my current plans.

iclementine avatar Jul 15 '24 08:07 iclementine

Hi @iclementine. Were you able to implement it? I will check on the configuration for AMD GPUs

radna0 avatar Aug 04 '24 22:08 radna0

Sorry for that, I was working on other projects and will be occupied recently.

iclementine avatar Aug 05 '24 07:08 iclementine