[diffusers] training is broken with xformers and PyTorch 2.1

[sayakpaul]

Related issue: https://github.com/huggingface/diffusers/issues/5368

When using PyTorch 2.1 and the latest stable build of xformers, our DreamBooth LoRA script for SDXL doesn't work. https://github.com/huggingface/diffusers/issues/5368 provides more details.

But when using SDPA in the same environment (i.e., no xformers), the issue seems to go away.

Dev environment for this can be found here: https://github.com/huggingface/diffusers/blob/main/docker/diffusers-pytorch-compile-cuda/Dockerfile

When using PyTorch 2.0.1 with xformers==0.0.21, there seem to be no issues with the exact same script. PyTorch was installed with pip install torch==2.0.1+cu117 --index-url https://download.pytorch.org/whl/cu117 inside a Docker image mounted from nvidia/cuda:11.7.1-cudnn8-runtime-ubuntu20.04.

Cc: @patrickvonplaten @williamberman

[ahmed1996said]

Facing the same issue here!

[linoytsaban]

Ran on colab using PyTorch 2.0.1 and xformers==0.0.21 and it still failed:

• diffusers version: 0.24.0.dev0
• Platform: Linux-5.15.120+-x86_64-with-glibc2.35
• Python version: 3.10.12
• PyTorch version (GPU?): 2.0.1+cu117 (True)
• Huggingface_hub version: 0.19.4
• Transformers version: 4.35.2
• Accelerate version: 0.24.1
• xFormers version: 0.0.21

[carlthome]

Same here but with AudioCraft

[soulteary]

try this https://github.com/facebookresearch/xformers/issues/960 build it from source code.