IPAdapter not compatible to torch.compile

[rootonchair]

Describe the bug

When use IPAdapter with torch.compile will return error message below:

torch._dynamo.exc.Unsupported: UNPACK_SEQUENCE NNModuleVariable()

Due to the use of this line

https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/embeddings.py#L879

Which can be fixed with a patch like:

    for l in self.layers:
        residual = latents

        comps = []
        for i in l:
            comps.append(i)
        [ln0, ln1, attn, ff] = comps

        encoder_hidden_states = ln0(x)
        latents = ln1(latents)
        encoder_hidden_states = torch.cat([encoder_hidden_states, latents], dim=-2)
        latents = attn(latents, encoder_hidden_states) + residual
        latents = ff(latents) + latents

Reproduction

from diffusers import AutoPipelineForText2Image
from diffusers.utils import load_image
import torch

pipeline = AutoPipelineForText2Image.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16).to("cuda")
pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="sdxl_models", weight_name=["ip-adapter-plus_sdxl_vit-h.safetensors"])
pipeline.set_ip_adapter_scale(0.6)
pipeline.unet.to(memory_format=torch.channels_last)
pipeline.unet = torch.compile(pipeline.unet, mode="max-autotune", fullgraph=True)

Logs

No response

System Info

• diffusers version: 0.28.0.dev0
• Platform: Linux-4.18.0-408.el8.x86_64-x86_64-with-glibc2.17
• Python version: 3.8.13
• PyTorch version (GPU?): 2.1.0+cu121 (True)
• Huggingface_hub version: 0.21.4
• Transformers version: 4.40.0.dev0
• Accelerate version: 0.28.0
• xFormers version: 0.0.22.post7
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

Who can help?

@sayakpaul @yiyixuxu

[sayakpaul]

Thanks for sending over the patch, too. Would you like to take a stab at sending over a PR?

[rootonchair]

Sure, PR on the way

[sayakpaul]

Closing with https://github.com/huggingface/diffusers/pull/7994