generative-models
generative-models copied to clipboard
Stability-AI
Question: Inpainting model for SDXL
Dear Stability AI Team
Thanks for an other great release!
Do you plan on releasing an inpainting model for SDXL too?
it will be v-prediction and zero terminal SNR, most likely. i think that will come with 1.1.
when to release the inpainting model？ there will be much use of the inpainting model
Thanks @bghira! Is there a timeline for this? Also, is the SDXL-base model v-prediction too or is that also in the 1.1 release?
SDXL can already be used for inpainting, see: https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/stable_diffusion_xl#inpainting
To me it seems like they are using the base model for inpainting.
Will there be a fine tuned version like stable-diffusion-2-inpainting?
Anyone notice inpainting with a mask alters the image the mask says it should not be altering? Is this expected?
@landmann If you are referring to small changes, than it is most likely due to the encoding/decoding step of the pipeline. We bring the image in a latent space (containing less information than the original image) and after the inpainting we decode it back to an actual image but in this process we are losing some information (the encoder is lossy as mentioned by the authors). This is expected behavior.
Workaround can be found here: https://github.com/huggingface/diffusers/issues/3514#issuecomment-1565221994
Is it possible to finetune SDXL 1.0 base model for inpainting?
I tried to do that but I get a error from UNET that it expects the input to have 4 channels but it got 9 channels (4 - input image, 4 - masked image, 1 - mask (B/W))
Am I doing anything wrong or should I wait for inpainting model release?
if you're feeling adventurous you can modify the pix2pix finetuner from Diffusers project.
it already uses 8 channel input.
new inpaint model https://huggingface.co/wangqyqq/sd_xl_base_1.0_inpainting_0.1.safetensors/blob/main/sd_xl_base_1.0_inpainting_0.1.safetensors
its also HALF so 16GB RTX nvidia doesnt work !
as i mentioned in webui post, this sdxl-inpaint model can work in 16-bit float and you don't need to appy --no-half-vae or --no-half. you can download the model, merge the code in pull request https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/14390 and try it!