Add StableDiffusion repaint pipeline

[nathanielherman]

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[nathanielherman]

Bump on this PR! Also @patrickvonplaten wdym by "leave it here"?

[anton-l]

Thanks for catching the issues @Randolph-zeng!

@nathanielherman let me know if you don't have bandwidth this week, I'd be happy to help getting the PR ready for merging :)

[nathanielherman]

Hey! I did most of the updates and will look at adding an integration test. On that note, AFAICT from here https://github.com/huggingface/diffusers/blob/main/tests/pipelines/stable_diffusion/test_stable_diffusion_inpaint_legacy.py#L352, the code is actually loading the non-legacy pipeline rather than the legacy one? I'm a bit confused how that's not breaking it though, since it's initializing it with a non-inpainting model ("CompVis/stable-diffusion-v1-4")

[Randolph-zeng]

Just curious, am I the only one that experienced the DDIM degradation here ? When I use the code in this PR I noticed that the DDIM almost failed completely in producing any meaningful impaint image that corresponds to the prompt. @nathanielherman Are you troubled by this same issue https://github.com/huggingface/diffusers/issues/1602 or does it work fine with you ? Thanks a lot if you can share some insight : )

[nathanielherman]

@Randolph-zeng hmm I'm confused by the linked issue, do you only get the issue for CFG outside of 6-7, or for any CFG? I only really use the default CFG of 7.5 but for that I get pretty reasonable outputs. (Though I wouldn't say repaint is obviously better results than default inpaint_legacy.)

[nathanielherman]

@anton-l bump on my question for the unit test, I just want to make sure I'm understanding correctly before I add my own unit test

[anton-l]

@nathanielherman regarding your question

the code is actually loading the non-legacy pipeline rather than the legacy one

The tests there carried over from the time when the legacy inpainting pipeline wasn't yet Legacy :) The pipeline loader substitutes the appropriate class for now: https://github.com/huggingface/diffusers/blob/31444f5790a8f91fc2caf3995fadc561e23fb279/src/diffusers/pipeline_utils.py#L539-L544 So you can safely assume that those tests are actually using StableDiffusionInpaintPipelineLegacy (we should probably update them, thanks for bringing it up!)

[nathanielherman]

red_cat_sitting_on_a_park_bench_repaint.npy.zip

[nathanielherman]

@anton-l makes sense! I've added the test and attached the npy file as a comment on this PR — IIUC from the docs, someone would need to upload that npy file and then I can update the test to download it from the url?

[anton-l]

@nathanielherman uploaded it to the repo: https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/repaint/red_cat_sitting_on_a_park_bench_repaint.npy But also feel free to set up a personal repository on the hub to link the files, we can adapt later! :)

Great progress on the PR, let me know if it's ready for the final review!

[nathanielherman]

@anton-l perfect, it should be good for final review now!

[patrickvonplaten]

Gently ping @anton-l for a final review

[patrickvonplaten]

Thanks for the nice PR @nathanielherman!

Three things from my side:

• 1. Could we also add fast tests here similar to: https://github.com/huggingface/diffusers/blob/f17fae641c7c268e60c479396891ea97a6b29aae/tests/pipelines/stable_diffusion/test_stable_diffusion_inpaint.py#L43
• 2. Could we also add docs as explained here: https://github.com/huggingface/diffusers/tree/main/docs
• 3. Could we remove all the deprecation messages?

Thanks!

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[Markus-Pobitzer]

Good Morning

Thanks for the great work. I am wondering why this pull request has been closed and how one can help.

[vlordier]

@anton-l is there something left we can do to merge this pipeline ?

[anton-l]

@vlordier the TODO is mostly just to update the tests as per the comments above (and fix any API issues uncovered by the common tests), and resolve the merge conflicts.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.