[Bug]: Soft Inpainting doesn't do anything: Automatic1111 Outperforms, Possible Forge Issue

[afrofail]

Checklist

• [X] The issue exists after disabling all extensions
• [X] The issue exists on a clean installation of webui
• [X] The issue is caused by an extension, but I believe it is caused by a bug in the webui
• [X] The issue exists in the current version of the webui
• [X] The issue has not been reported before recently
• [ ] The issue has been reported before but has not been fixed yet

What happened?

I recently came to an understanding while following a tutorial on inpainting that featured the "Soft Inpainting" option. Despite adhering to the instructions and utilizing the "Soft Inpainting" checkbox within Forge, it didn't perform as expected—especially when I increased the mask blur, which what was demonstrated in the video using the latest Automatic1111 fork. The inpainting results from Forge tended to be more blurred overall. In contrast, I've found that Automatic1111 provides much better outcomes, although Forge operates significantly faster and without VRAM errors. I would be immensely appreciative if this issue could be addressed. I applied the same "Soft Inpainting" settings on both platforms.

Steps to reproduce the problem

What should have happened?

When enabling "Soft Inpainting" it should preserve the masked selection and inpainting.

What browsers do you use to access the UI ?

Google Chrome

Sysinfo

sysinfo-2024-03-05-10-43.json

Console logs

None

Additional information

No response

[catboxanon]

Haven't tested to verify myself yet, but taking a cursory look at the infotext in the screenshots, you're using Euler a in both UIs, but in Forge, your eta noise seed delta (ENSD) value is set to 31337. Because you're using an ancestral sampler (the "a" part), that's very likely going to make a difference. If you use a non-ancestral sampler like Euler instead does the issue still occur?

[afrofail]

Haven't tested to verify myself yet, but taking a cursory look at the infotext in the screenshots, you're using Euler a in both UIs, but in Forge, your eta noise seed delta (ENSD) value is set to 31337. Because you're using an ancestral sampler (the "a" part), that's very likely going to make a difference. If you use a non-ancestral sampler like Euler instead does the issue still occur?

Thanks for noticing, unfortunately I get the same problem after changing the ENSD to -1. As you can see above, results are different, the "soft inpainting" feature in Automatic1111 seems more stable right now, especially when using SDXL models.

[catboxanon]

Could you upload the original image, mask, and generated images here for reproducibility purposes?

[afrofail]

[afrofail]

ENSD

Changed ENSD to 0 and still have the same issue between the two.

[catboxanon]

So it looks like this doesn't work as intended at all due to a large portion of the CFGDenoiser's forward function having been rewritten, and the relevant mask blending code being gutted out.

https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/eee46a5094f748ebe9fe4978b0d440a8ebbab4a6/modules/sd_samplers_cfg_denoiser.py#L168-L178

I think @lllyasviel needs to take a look at this one.

Edit: May have a fix soon -- wait warmly.

[catboxanon]

@afrofail Can you let me know if https://github.com/lllyasviel/stable-diffusion-webui-forge/pull/494 resolves this for you? It's not a complete solution as it undoes some changes by lvmin, but it would make this otherwise work.

[afrofail]

@afrofail Can you let me know if #494 resolves this for you? It's not a complete solution as it undoes some changes by lvmin, but it would make this otherwise work.

I just replied to the new commit merge but I'll post again here!

This fixed the issue, I can't thank you enough for figuring this out.

Forge

Automatic1111

[youlijie]

How do I apply it ? I changed the code manually like the commit but still do nothing for me.