ml-stable-diffusion icon indicating copy to clipboard operation
ml-stable-diffusion copied to clipboard

Published 20 hours ago • verified publisher icon apple

Inpainting support

Open aajank opened this issue 1 year ago • 4 comments

does this support inpainting or do we have to wait further ?

aajank avatar Mar 28 '23 10:03 aajank

+1 to this

I see there was some plans for it already, but commented out? Is anyone actively working on it?

https://github.com/apple/ml-stable-diffusion/blame/48f07f24891155a14c51dd835bba7371bdf32d0e/swift/StableDiffusion/pipeline/StableDiffusionPipeline.Configuration.swift#L14

ZachNagengast avatar Jun 30 '23 21:06 ZachNagengast

I have read that in the python-diffusers world, inpainting works best with dedicated models that have an additional layer (in the Unet?) to directly accept the inpaint mask data. This helps preserve the mask edge details. I wonder if the ControlledUnet we now have also has an extra layer, to accept the ControlNet input, and if inpainting could leverage this, and be much simpler to implement now than it was before ControlNet?

jrittvo avatar Jun 30 '23 21:06 jrittvo

Looks like this PR added hole punching support, allowing a ControlNet model to do inpainting. More info here https://github.com/godly-devotion/MochiDiffusion/pull/272

ZachNagengast avatar Jul 11 '23 02:07 ZachNagengast

It works fine. ‎SD15ControlNet11InPaint ‎001

ynagatomo avatar Jul 13 '23 14:07 ynagatomo