Train text to image slower with xformers

[gltanaka]

Describe the bug

The train text to image is slower when xformers 0.0.16 is enabled (6 hours and 4 mins) vs when x formers is not used (5 hours and 45 mins) on a A100 40GB GPU. About the same amount of memory (20 GB) is used in each case.

Reproduction

Comment out uunet.enable_xformers_memory_efficient_attention()

using fp16

Logs

No response

System Info

Python 3.9

[pcuenca]

Hi @gltanaka! We have reports from people having trouble training with xformers 0.0.16: https://github.com/huggingface/diffusers/issues/2234#issuecomment-1416931212. Could this be the same thing that's happening to you?

[gltanaka]

I tried pip install --pre xformers==0.0.17.dev461 It about the same runtime BTW, I am also using deepspeed 0. It is about the same runtime without deepspeed.

[gltanaka]

Here is my nvidia-smi: Mon Feb 20 08:07:29 2023
+-----------------------------------------------------------------------------+ | NVIDIA-SMI 525.60.13 Driver Version: 525.60.13 CUDA Version: 12.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA A100-SXM... Off | 00000000:00:04.0 Off | 0 | | N/A 40C P0 122W / 400W | 17799MiB / 40960MiB | 40% Default | | | | Disabled | +-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 30026 C ...mples-p1GFpC0G/bin/python 17796MiB |

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.