Add PAG Support for Stable Diffusion Inpaint Pipeline

[darshil0805]

What does this PR do?

Fixes #8710 Partially

Before submitting

• [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• [x] Did you read the contributor guideline?
• [x] Did you read our philosophy doc (important for complex PRs)?
• [ ] Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• [ ] Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
• [x] Did you write any new necessary tests?

Who can review?

@yiyixuxu Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

Hi, I have written the code and added the relevant tests and ran them, they failed in 3 cases, attached below, I'll need some assistance to test the code using images as I am GPU poor

[darshil0805]

Hi @yiyixuxu @a-r-r-o-w, following up in case you missed this, let me know if there is anything I can do to speed up this process

[asomoza]

Hi, can you please resolve the conflicts?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[darshil0805]

from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image
import torch

device = "cuda"

# Load the image-to-image pipeline with pre-trained weights
pipe = AutoPipelineForInpainting.from_pretrained("runwayml/stable-diffusion-inpainting", enable_pag=True, torch_dype = torch.float16)
pipe.to(device)
pipe.enable_model_cpu_offload()


img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"

init_image = load_image(img_url).convert("RGB")
mask_image = load_image(mask_url).convert("RGB")

prompt = "A majestic tiger sitting on a bench"


pag_scales =  [0.0, 3.0]
guidance_scales = [7.5]

grid = []
for pag_scale in pag_scales:
    for guidance_scale in guidance_scales:
        generator = torch.Generator(device="cpu").manual_seed(1)
        images = pipe(
            prompt=prompt,
            image=init_image,
            mask_image=mask_image,
            strength=0.8,
            num_inference_steps=50,
            guidance_scale=guidance_scale,
            generator=generator,
            pag_scale=pag_scale,
        ).images
        images[0]

        grid.append(images[0])

# save the grid
from diffusers.utils import make_image_grid
make_image_grid(grid, rows=len(guidance_scales), cols=len(pag_scales)).save("test_out.png")

I was able to get this code running on Colab, this is how the outputs look:

[yiyixuxu]

I think CI still not passing can you run make style?

[darshil0805]

I think CI still not passing can you run make style?

I've ran it and made the change, hopefully it passes now

[yiyixuxu]

I'm still seeing an import error https://github.com/huggingface/diffusers/actions/runs/11266431877/job/31368980510?pr=9386#step:5:250

[darshil0805]

I'm still seeing an import error https://github.com/huggingface/diffusers/actions/runs/11266431877/job/31368980510?pr=9386#step:5:250

Not sure why there is an import error, I'm able to test the script through Colab Here's the notebook: https://colab.research.google.com/drive/1wgqDCNDlX6Fb47BMQN7GzWKtvhpounXD?usp=sharing

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[yiyixuxu]

can anyone help look into the import error here?

[ParagEkbote]

Can I please contribute to fixing the import errors?

cc: @yiyixuxu

[yiyixuxu]

@hlky feel free to merge if it looks good to you!