[Feature Request]: UniPC img2img & inpainting support

[ghost]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

It would be nice to be able to do img2img and inpainting with this sampler. Its really fast and the details are pretty good in lower step levels.

I would have done it myself but I am not really an experienced coder, so it would take a very long time for me.

Proposed workflow

Its in the description

Additional information

No response

[icepeng]

I'm looking into this issue, basically UniPC sampler does not implement make_schedule, stochastic_encode, decode method. However, img2img script implemented in UniPC repo calls those methods, which is confusing.

[EllangoK]

The img2img script uses either the DDIM or PLMS sampler based on options, even though it is in https://github.com/wl-zhao/UniPC repo. It basically silently switches to DDIM which is what is done here as well pretty much.

https://github.com/wl-zhao/UniPC/blob/63decc45a2206a2868381075bb9cef09cd931a9e/example/stable-diffusion/scripts/img2img.py#L19-L20

https://github.com/wl-zhao/UniPC/blob/63decc45a2206a2868381075bb9cef09cd931a9e/example/stable-diffusion/scripts/img2img.py#L205-L209

Doesn't mean it is not possible, in fact according to the issue created in there repo it is possible with diffusers and huggingface.

[icepeng]

Silent switching works well for hires.fix.

However, certain scripts such as ddetailer are experiencing issues because they are utilizing StableDiffusionProcessingImg2Img with the same sampler internally.

Would it be possible to integrate the same switching logic within processing.py/StableDiffusionProcessingImg2Img?

I attempted to implement make_schedule, stochastic_encode, and decode within UniPCSampler, but the resulting image was excessively blurry.

[EllangoK]

Same I'm messing around with it and pretty much did probably what you did. According to this https://github.com/wl-zhao/UniPC/issues/6#issuecomment-1464911620 though there should be just native support and we shouldn't need to modify processing.py/StableDiffusionProcessingImg2Img?

I am looking at the diffusers implementation of it right now to see how they did it.

[icepeng]

Same either, looking into diffusers implementation https://github.com/huggingface/diffusers/pull/2373/files#diff-70d67333f2a5d08d4529ba96b0f6e6c6c8a73d41729a89b670d72d2515671483

implementing add_noise and step in current UniPCSampler might work?

[EllangoK]

Holdup, @icepeng did you try the demo on hugging face img2img? I just tried it now and it doesn't work for me at all. Txt2img does work fine

I actually just think it doesn't work for diffusers either and perhaps the owners of UniPC did not realise that when they made their comment.

[icepeng]

It errors out for me too. I think we have to keep discuss on https://github.com/wl-zhao/UniPC/issues/6#issuecomment-1464911620

[ghost]

So, no progress so far? I had my doubts that this will work out. It would be such a win for animations, to do 2 or 3 steps or even 1 and get away with it in terms of coherency and quality

edit: I am too stupid to do it, so Im just... kindly asking

[ImpossibleAd436]

At the moment it is still using DDIM for hires fix, and using UniPC for inpainting and Img2Img isn't an option.

Is there some technical limitation of the sampler which doesn't allow for it to be used for these things? I understand that as it stands it doesn't implement decode, but will it ever be possible?

It is by far the best sampler I have used and it would be amazing to be able to use it for inpainting also. Thanks.

[barasabwb]

Any progress on this?

[ImpossibleAd436]

Any update on this? Will it ever happen? It's really a shame to not be able to inpaint with this sampler I hope someone can add to it what is neccessary.

[catboxanon]

Supported as of https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/8285a149d8c488ae6c7a566eb85fb5e825145464