Regularize Latent input for SD and SDXL img2img

[AI-Casanova]

I am currently trying to implement seed interpolation in SDNext. This can be done smoothly for SD and SDXL pipelines but not for img2img

Currently latents can be passed to StableDiffusionXLImg2ImgPipeline but they are promptly ignored here https://github.com/huggingface/diffusers/blob/8974c50bff1f51c5a1208723af88d4b4ed7f2b16/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_img2img.py#L1331

I could work around that by setting denoising_start and passing a noised image. However SD img2img has no way to avoid adding noise without hijacking the entire function https://github.com/huggingface/diffusers/blob/07349c25fe1e2762b562207eedc7ae28379b091d/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_img2img.py#L713

Proposed Solution: Add latents kwarg to StableDiffusionImg2ImgPipeline and check for None before calling the prepare_latents function.

[yiyixuxu]

Hi:

do you want to open a PR?

[AI-Casanova]

Might not be very soon, but I might just do that.

[nicollegah]

I'd love a solution for this too!

[nileshkokane01]

@yiyixuxu ,

If I understood correctly.

check for `latents':

1. If not None bypass prepare_latents
2. Else prepare_latents as its currently now .

would the above solution suffice?

[bghira]

bypassing it when we use latents input might be fine, but it will change behaviour for users who expect it to operate as it is

[nileshkokane01]

Shouldn't that be documented then?

Because when you expect to work as is, the passed latent are overridden anyway.

[vladmandic]

bypassing it when we use latents input might be fine, but it will change behaviour for users who expect it to operate as it is

sorry, this makes no sense - it allows latents to be passed as param, but then ignores it. if it changes broken behavior, its pretty much a fix, not even a new feature. and yes, fixes sometimes change behavior if user is relying on broken behavior.

i hope we're not going towards so-called "bug-compatible" levels :)

[nileshkokane01]

check for `latents':If not None bypass prepare_latents Else prepare_latents as its currently now .

I think then the above holds good. Let me know otherwise . I can take this up for PR.

[yiyixuxu]

@nileshkokane01 hey please open a PR :)

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.