Proposal: Howto speedup Stable Diffusion by 30%

[oss-roettger]

First of all, thank you for making SD publicly available! I’d like to give back a little.

While evaluating SD I found out that there is an additional way to speedup image generation - besides less ddim_steps: you can exit the sampling loops in DDIMSampler.ddim_sampling() and DDIMSampler.decode() earlier if you use a intermediately predicted latent 'pred_x0' for the image generation instead of the final latent. The predicted latent is quite stable after 70% of the ddim_steps. This saves upto 30% of time or energy, so it would be a nice feature to be implemented for a new preview mode or for large batch processing.

I haven’t opened a pull request since to many files would be effected to pass through a preview switch, but I described and illustrated the idea and 4 lines of code hack to try out that modification here: https://github.com/oss-roettger/stable-diffusion-speedup

[Marcophono2]

That sounds really interesting! I will try this out later today. Thank you for that great input! I think you are accidently in that code part which I pointed to in my own issue here. It is directly posted before yours. May be you have also an idea or a solution for this? -> https://github.com/CompVis/stable-diffusion/issues/317

I can return a lot of values (noise_pred) but that are only the values for each pixel. So at least there is a (matrix-) multiplication needed but may be I am completely at the wrong end here.

Best regards Marc