[Pipeline] AnimateDiff SDXL

[a-r-r-o-w]

What does this PR do?

Fixes #6158.

Attempt at integrating https://github.com/guoyww/AnimateDiff/tree/sdxl.

Relevant discussion: https://github.com/huggingface/diffusers/pull/5928#issuecomment-1851737456

Continuation of #6195. Unfortunately, the code base from current main and the one in 6195 felt really divergent. I ended up messing my local branch. I thought starting fresh is a better idea since we had many unwanted changes in the original PR. This is still a WIP.

Thanks to ModelsLab for providing GPU support for faster testing.

Before submitting

• [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• [x] Did you read the contributor guideline?
• [x] Did you read our philosophy doc (important for complex PRs)?
• [x] Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• [x] Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
• [x] Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

@DN6 @sayakpaul @guoyww

[a-r-r-o-w]

Link to Colab.

Target usage

import torch
from diffusers import AnimateDiffPipeline, DDIMScheduler, EulerDiscreteScheduler, DEISMultistepScheduler
from diffusers.models import MotionAdapter
from diffusers import AnimateDiffSDXLPipeline

adapter = MotionAdapter.from_pretrained("a-r-r-o-w/animatediff-motion-adapter-sdxl-beta", torch_dtype=torch.float16)

# model_id = "stabilityai/stable-diffusion-xl-base-1.0"
model_id = "stablediffusionapi/dynavision-xl-v0610"
# model_id = "Lykon/dreamshaper-xl-1-0"

# scheduler = EulerDiscreteScheduler.from_pretrained(
# scheduler = DEISMultistepScheduler.from_pretrained(
scheduler = DDIMScheduler.from_pretrained(
    model_id,
    subfolder="scheduler",
    clip_sample=False,
    timestep_spacing="linspace",
    beta_schedule="linear",
    steps_offset=1,
)
pipe = AnimateDiffSDXLPipeline.from_pretrained(
    model_id,
    motion_adapter=adapter,
    scheduler=scheduler,
    torch_dtype=torch.float16,
    variant="fp16",
).to("cuda")

pipe.enable_vae_slicing()
pipe.enable_vae_tiling()

result = pipe(
    prompt="a panda surfing in the ocean, realistic, hyperrealism, high quality",
    negative_prompt="low quality, worst quality",
    num_inference_steps=20,
    guidance_scale=8,
    width=1024,
    height=1024,
    num_frames=16,
)

from diffusers.utils import export_to_gif
export_to_gif(result.frames[0], "animation.gif")

SDXL Base
DynaVisionXL
DreamshaperXL

Still experimenting and trying to find if I missed something. Quality seems to be okay-ish but definitely need to look for better parameters/models. SDXL checkpoint by guoyww is still a beta release, and maybe we could wait for official release before considering merge.

[a-r-r-o-w]

It'd be great to support the SDXL version but since there are no official checkpoints or a training script, it makes it harder for the community to experiment with and does not seem like a good feature to add in core diffusers. I will try cooking a motion adapter training script based on what I understand so far when I find the time, and try reaching out to the authors over other mediums.

cc @guoyww @limbo0000 @AnyiRao @wyhsirius Thank you for your amazing work! Any suggestions and updates would be awesome :heart:

[yiyixuxu]

@a-r-r-o-w @DN6 does that mean we want to add to the community folder for now? what's the plan?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[a-r-r-o-w]

@a-r-r-o-w @DN6 does that mean we want to add to the community folder for now? what's the plan?

It's a bit of a difficult situation since no official checkpoints exist. The pipeline can be moved into community but the modelling changes here are more general and add functionality without breaking support for existing motion adapters (I hope, since I'm yet to test properly. The currently failing tests are only specific to what I added which is a positive), so I think that should be okay. The changes to the UNetMotionModel essentially just replicate the logic of UNet2dConditionModel i.e. there's nothing new in modelling parts as such. Let me know your thoughts and I'll adapt the code accordingly, thanks.

[DN6]

@a-r-r-o-w @DN6 does that mean we want to add to the community folder for now? what's the plan?

We are seeing some traction on the Motion Adapters. So I it would be really nice to have this in our core pipelines. Unfortunately, as @a-r-r-o-w mentioned, the checkpoint is still technically in beta, so perhaps community is the best place for the moment. That way people can still use the pipeline/model with Diffusers.

Let me see if I can get in touch with the authors about their plans on an official release.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sayakpaul]

Cc: @DN6 could you share any progress on https://github.com/huggingface/diffusers/pull/6721#issuecomment-1969147261? And this is definitely not stale I think?

[a-r-r-o-w]

Sharing a result with FreeInit enabled for same conditions with Dreamshaper XL 1.0 after the latest changes:

without freeinit	with freeinit

This should be mostly ready for a merge I believe if we're okay with the beta checkpoints. I'm yet to test if the SD1.5 models break with the changes to UNetMotionModel but ideally everything should work since no old tests failed regarding it.

[a-r-r-o-w]

Just tested SD1.5 models with the changes to UNetMotionModel. Everything seems to be working for both SD and SDXL. So I think this is very close to completion. @DN6 The sdxl motion adapter checkpoint in diffusers format is available on my account with about ~500 downloads so far. Will you be able to move it to the authors' accounts or are we okay with this? I'll update the example code accordingly.

[DN6]

@a-r-r-o-w I think it's fine to have it on your account since you worked on the conversion.

[sayakpaul]

@DN6 the checkpoints are better in their original organizations actually. The model card can contain a note about Aryan's contributions but they should ideally reside under the original org/author.

[DN6]

@DN6 the checkpoints are better in their original organizations actually. The model card can contain a note about Aryan's contributions but they should ideally reside under the original org/author.

Cool in that case we'll try to get in touch with the authors and move the checkpoints @a-r-r-o-w. We can cite your work in the model card.

[a-r-r-o-w]

Sounds good to me. It was only a few line changes to the already existing motion adapter script and not much work so it doesn't really matter :)

FAILED tests/pipelines/animatediff/test_animatediff_sdxl.py::AnimateDiffPipelineSDXLFastTests::test_save_load_optional_components - AttributeError: 'NoneType' object has no attribute 'tokenize'

All failing tests fixed with the latest commit locally. Seems like the above was handled differently for SDXL so I've copied over the logic from SDXL tests.

[a-r-r-o-w]

Cool in that case we'll try to get in touch with the authors and move the checkpoints @a-r-r-o-w. We can cite your work in the model card.

Any updates from them @DN6?

[DN6]

Hi @a-r-r-o-w Yes we'll move it.

We can just transfer the checkpoint from your org to theirs. Is everything ready on your end?

Could you prep the model card and add something along the lines of "Converted to Diffusers by @a-r-r-o-w" You can link it to your preferred profile (GitHub or the Hub) so that your work is also attributed.

[a-r-r-o-w]

@DN6 Hey! I believe everything should be good for transfer hopefully. Please find the checkpoint here.

[DN6]

@a-r-r-o-w Just opened a PR on your model repo to update the code snippet in the model card and in the docs in this PR. I think we're ready to go once those changes are made. 👍🏽