RePaintPipeline always returns black image when pipe.to("mps") is called

[GaidamakUA]

Describe the bug

When I try to do the inpainting with RePaintPipeline on the MPS backend it only returns black image. It works fine with CPU, but very slow.

Reproduction

scheduler = RePaintScheduler.from_config("google/ddpm-ema-celebahq-256")
pipe = RePaintPipeline.from_pretrained("google/ddpm-ema-celebahq-256", scheduler=scheduler)
device = torch.device("mps")
pipe = pipe.to(device)
def inpaint(image, mask):
    output = pipe(
        original_image=image,
        mask_image=mask,
        num_inference_steps=20,
        eta=0.0,
        jump_length=1,
        jump_n_sample=2,
        return_dict=False,
    )
    print(output)
    image.paste(output[0][0], (x, y))
    return np.array(image)

Logs

No response

System Info

• diffusers version: 0.7.2
• Platform: macOS-12.6-arm64-arm-64bit
• Python version: 3.9.12
• PyTorch version (GPU?): 1.13.0.dev20221002 (False)
• Huggingface_hub version: 0.10.0
• Transformers version: 4.24.0
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: no

[patrickvonplaten]

Interesting! We did some bigger changes for cycle diffusion on "main" - could you maybe try again?

[patrickvonplaten]

Also maybe cc @pcuenca

[pcuenca]

Hi @GaidamakUA, thanks for reporting this.

It doesn't work for me either, but it does not return black images. This is what I get on mps for the image and mask taken from this example:

For reference, this is the result when running on cuda (not perfect, I just ran it for a few steps):

What's also puzzling is that the unmasked image area to be preserved (the bottom half of the face) appears to have inverted colors in the mps version.

Is this the same thing you are seeing? Also, do you know if this ever worked in a previous version of diffusers?

[GaidamakUA]

Still black

• diffusers version: 0.8.0.dev0
• Platform: macOS-12.6-arm64-arm-64bit
• Python version: 3.9.12
• PyTorch version (GPU?): 1.13.0.dev20221002 (False)
• Huggingface_hub version: 0.10.0
• Transformers version: 4.24.0
• Using GPU in script?:
• Using distributed or parallel set-up in script?: Input Mask Output

[GaidamakUA]

Hi, @pcuenca, any chances for a fix?

[patrickvonplaten]

@anton-l could you take a look?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[patrickvonplaten]

Gently ping @pcuenca @anton-l, but also please note @GaidamakUA that our MPS integration is still very much experimental as it was only recently supported in PyTorch and RePaint is also not the most used pipeline I believe :sweat_smile:

[pcuenca]

Yeah, it's in my to-do list but I still haven't found the time to dive deeper after the initial assessment I made when @GaidamakUA reported the problem. Unfortunately I'm fully booked up until the 15th, will come back to this then. Sorry about that!

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[pcuenca]

~~Maybe related: https://github.com/invoke-ai/InvokeAI/commit/bd57793a658a86d5d81f86401e2d90ec10a92046, https://github.com/pytorch/pytorch/issues/91617, https://github.com/kulinseth/pytorch/pull/222~~

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.