huggingface
Add support for Outpainting with SDXL
SDXL for Outpainting functionality can be improved. Attached are some attempts at Outpainting with StableDiffusionXLInpaintPipeline and stabilityai/stable-diffusion-xl-base-1.0 checkpoint.
Given this image/mask combination ((black pixels ignored and white pixels are to be filled)
and the prompt A girl sitting in a library full of books, the pipeline produces the following images
The pipeline is filling in details in the masked area that is related to the prompt, but doesn't seem to make any use of the image provided (the masked area is filled independently of the image content). This is likely due to the fact that the base SDXL UNet is not conditioned on outpainting masks and the pipeline relies on an approximation to perform inpainting/outpainting:
https://github.com/huggingface/diffusers/blob/87ae330056f6942817656c8f7146283e90cf986b/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_inpaint.py#L1318-L1328
A UNet that accepts Mask Latents in the channels dimension should fix the issue.
I've attached a Colab Notebook here to test Outpainting: https://colab.research.google.com/gist/DN6/a94626ac9e062f2ff1cc31b8bc2f72ed/sdxl_outpainting.ipynb
The above example is an inpainting pipeline, but a non-inpainting ckpt/weights, so this is the results I would expect. Have you tried diffusers/stable-diffusion-xl-1.0-inpainting-0.1 ?
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
If you use the checkpoint reimager mentions you can get results like this! Thanks reimager.
@relh what is reimager? can you share a code example how do you get this result?
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.