Reverse sampling for schedulers

[neverix]

Model/Pipeline/Scheduler description

Some of the samplers (e.g. DDIM) are fully reversible. A simple API addition of reverse_step() or a mixin can be implemented. This will allow interpolating between images not generated by diffusion models.

Open source status

• [X] The model implementation is available
• [ ] The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

https://github.com/crowsonkb/v-diffusion-jax/blob/master/diffusion/sampling.py by @crowsonkb implements reverse sampling for V-objective diffusionon

[patrickvonplaten]

Hey @neverix,

Could you maybe write down a code example with diffusers API that shows that behavior (or open a PR maybe?). This would really help us with quickly seeing the necessary changes

[neverix]

Yep, here's a notebook with the draft for the implementation

[patrickvonplaten]

Sorry, I still don't fully understand the use case. The images that is output after:

decoded_latents = latents.clone()
with autocast("cuda"), inference_mode():
    for i, e in enumerate(tqdm(ddim.scheduler.timesteps)):
        decoded_latents = ddim.scheduler.reverse_step(ddim.unet(decoded_latents, e).sample, e, decoded_latents).next_sample
show_lat(decoded_latents)

is very blurred -> it doesn't seem like the reverse_step function works correctly here?

[neverix]

Haha, I forgot to flip the timesteps in the version I saved. It works fine if you do.

[patrickvonplaten]

Could you maybe updated the google colab to reflect this change (or even better), post the required code here so that we can dive into it / reproduce it ? :-)

Thanks a lot !

[neverix]

The code for this is in the Colab linked above

[neverix]

https://arxiv.org/abs/2210.05559 has since come out which names reverse sampling "DPM-Encoder"

[patrickvonplaten]

Very cool! @neverix do you think it might be possible to just add this as a new stand-alone scheduler / sampler? If you want we could open a PR to add this DPM-Encoder-Scheduler here: https://github.com/huggingface/diffusers/tree/main/src/diffusers/schedulers ? What do you think? :-)

[neverix]

For just DDIM this would work - but the code is pretty much the same between the forward and reverse process, so I think inheriting or adding it as a method for the DDIM scheduler would work better. I'll implement the rest of the paper in the notebook and write back

[neverix]

#885's functionality now works. Will work on the style, any suggestions for the functionality?

[patrickvonplaten]

Cool will review the PR :-)

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.