[Feature Request]: MAT Outpaint

[ttuleyb]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

Implement this to make outpainting much better: Hugging Face SD Mat Outpainting

I think this will benefit outpaint a lot here is why: It seems that most "bad" outpaintings seem to decide the wrong way when denoising at the start, this leads to the denoising creating objects or patterns that don't quite match the original image and look wrong/cropped. This proposed solution quickly creates an estimate of the outpainting, getting the colors quite accurately, then we diffuse the rest of the image from that essentially doing img2img to our noise given by the Mat Outpaint.

This significantly improves outpainting quality as it eliminates cropping (unless the MAT outpaint has screwed up which happens way less than SD screwing up with the initial few steps of denoise) and Stable Diffusion is much better at outpainting when the patterns are already kinda there compared to pure random noise.

Proposed workflow

1. Go to img2img
2. Press Script
3. Select "Mat Outpainting"
4. Take settings from poorman's outpainting + add settings from the demo Hugging Face SD Mat Outpainting
5. When "Generate" is pressed, the mat outpainting creates the rough outpaint with the mask, this mask and rough outpaint is then fed to img2img so that SD diffuse the outpainted section fully rather than leaving a rough "draft" outpaint.

Additional information

This may also be implemented in the form of a script/extension.

[Ehplodor]

This could be used as a primer for inpainting, too

[kalle07]

you tryed it it with an SDXL-inpaint model ?

btw: try fooocus for outpaint ... best ever ;) https://github.com/lllyasviel/Fooocus