[Bug]: img2img Inpaint upload generates erroneous images with the mask

[grummelgrisgram]

trafficstars

Is there an existing issue for this?

• [x] I have searched the existing issues and checked the recent builds/commits

What happened?

There seems to be a bug in the "Inpaint upload" feature of img2img in combination with a mask which is not completely white. In short, the mask seems to drastically influence the generated content, even the content which is only located on the pure white part of the mask. The white part of the mask should not influence the content though, e.g. the content should still be unrelated to source image with "Denoising Strength" parameter set to 1. But it isn't as depicted below. In general, I should still be able to influence the content, it should not (nearly) only be influenced by the present image if a (not only white) mask is set.

Steps to reproduce the problem

For demonstration purposes, I used "Denoising Strength" = 1 here, so I'd always expect to get a completely "unrelated image" at the part of the image where only white mask pixels are located.

a) If no mask is used (or only white mask) together with "Denoising Strength" set to 1, simply the whole image should be generated unrelated to the source image. This works as expected as you can see here:

b) If a mask is used with a black gradient border together with "Denoising Strength" set to 1, the part of the image with white mask should be generated unrelated to the source image. The border should be faded accordingly. This works not as expected as you can see here: Please note, that the border is not touched resp. faded (which is working as expected) and the part of the image with white mask is modified. But this is by no means a "Denoising Strength" of 1 here. Instead, it is only modified in a very minor way! I'd expect something similar to (a) here instead.

Since I only changed the mask between case (a) and (b), I think something is going wrong here...

By the way, the issue is gone with only white mask, similar to no mask at all:

Example source image and mask:

What should have happened?

With "Denoising Strength" of 1, the image should always be unrelated - apart from the black parts of the mask of course.

Commit where the problem happens

ea9bd9fc

What platforms do you use to access the UI ?

Windows

What browsers do you use to access the UI ?

Google Chrome, Brave

Command Line Arguments

--listen --enable-insecure-extension-access --xformers --api

List of extensions

No

Console logs

################################################################
Install script for stable-diffusion + Web UI
Tested on Debian 11 (Bullseye)
################################################################

################################################################
Running on user user
################################################################

################################################################
Repo already cloned, using it as install directory
################################################################

################################################################
Create and activate python venv
################################################################

################################################################
Launching launch.py...
################################################################
Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0]
Commit hash: ea9bd9fc7409109adcd61b897abc2c8881161256
Installing requirements for Web UI

Launching Web UI with arguments: --listen --enable-insecure-extension-access --xformers --api
Loading weights [c6bbc15e32] from /home/user/stable-diffusion-webui/models/Stable-diffusion/sd-v1-5-inpainting.ckpt
Creating model from config: /home/user/stable-diffusion-webui/configs/v1-inpainting-inference.yaml
LatentInpaintDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.54 M params.
Applying xformers cross attention optimization.
Textual inversion embeddings loaded(0):
Model loaded in 23.4s (load weights from disk: 4.8s, create model: 1.6s, apply weights to model: 13.2s, apply half(): 0.9s, load VAE: 2.1s, move model to device: 0.6s).
Running on local URL:  http://0.0.0.0:7860

To create a public link, set `share=True` in `launch()`.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:06<00:00,  2.97it/s]
Total progress: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:05<00:00,  3.59it/s]

Additional information

No response

[Vector-Cross]

in my case, any inpaint seems doesn't work

[rakataprime]

@grummelgrisgram , in your first screen shot you have inpaint area set to full picture. Wouldn't that not have a sharp inpainting boundary? What happens with inpaint area set to inpaint only masked? I have noticed that without the inpaint only masked set the final inpainting output often lose the inpainting effect.

[Vector-Cross]

@grummelgrisgram , i've solved mine. Just make sure the gradio version is as same the requriments.txt.

[grummelgrisgram]

@rakataprime, as I wrote, the first screenshot is working as expected. Setting the inpaint area to "Only masked" does not change anything for me. It's behaving the same way as "Whole picture".

@Vector-Cross: Well, no. Also your screenshot is showing a wrong result. If I set "Denoising strength" to 1, I'd expect a completely different result at the white part of the mask. Your result shows a fish, yes. But in the background, I still see lava and smoke although not specified in the prompt. This would be okay if "Denoising strength" is set to something lower than 1 but not while it is set to 1.

I tested it again but again the same behavior as I described above. (Not surprisingly because git source did not change at master branch since then.)