Implement UniPC sampler

[space-nuko]

Describe what this pull request is trying to achieve. Implement a new sampler, recently released here: https://github.com/wl-zhao/UniPC Related: #7705

Additional notes and description of your changes

I adapted the code from their Stable Diffusion example. Some changes needed to be made to support multicond.

Environment this was tested in

• OS: Windows
• Browser: chrome
• Graphics card: NVIDIA RTX 3090

Screenshots or videos of your changes

[ClashSAN]

(non-cherrypicked) 8 steps comparison - 512x768

5 steps comparison - 512x512

It's the same speed as DPM++ 2M Karras looks like the superior sampler at low steps.

[hithereai]

Looking great man!

Edit: 28 it/s on a 4090, where I get around 34 with euler_a. This is insane considering the fast resolving.

Edit: Indeed doesn't work with v2.1-768. But it does work with the 512 v2.1 model.

[Skeula]

I tried this out and it's shocking how fast it is, even separate from needing fewer steps

[yohm42]

Is this supposed to work with 2.x? Here's a sample I'm getting

a flower Steps: 10, Sampler: UniPC, CFG scale: 7, Seed: 0, Size: 768x768, Model hash: dcd690123c, Model: v2-1_768-ema-pruned, Eta DDIM: 1

[JDadders]

I noticed the commit message about this sampler not supporting img2img yet. Anyone have any insight as to why that is and if it might be resolved?

[space-nuko]

I noticed the commit message about this sampler not supporting img2img yet. Anyone have any insight as to why that is and if it might be resolved?

Same reason as why PLMS doesn't support img2img as far as I understand, those samplers are missing some methods used by img2img

[uservar]

This seems like a powerful sampler already, but I think we could make it work better with webui by further changing the example sampler.py from the original repo.

Some ideas:

• Consider using a different file instead of changing sd_samplers_compvis.py, maybe include the code from sampler.py so it's in one file where we can make changes
• Check if self.model.parameterization == "v" to determine model_type, either "v" if it's true or "noise" (this would support SD2.X 768-v models)

[space-nuko]

Thanks for the feedback, I would never have thought of the SD2.X check you mentioned

I edited sd_samplers_compvis.py because the original repo states you can replace the DDIMSampler with UniPCSampler as a drop-in so I felt it made sense to group UniPC with DDIM, also a bit of code is shared there for reconstructing multicond and it's almost exactly the same between UniPC and DDIM

[space-nuko]

And here's what the above prompt looks for me now

a flower
Steps: 10, Sampler: UniPC, CFG scale: 7, Seed: 2496947002, Size: 768x768, Model hash: ad2a33c361, Model: v2-1_768-ema-pruned, ENSD: 31337

[fractal-fumbler]

should it work with --no-half only?

[sALTaccount]

From my understanding of the paper, UniPC can be applied to all DPM based samplers. Would it maybe make more sense to have this as some sort of a check box rather than as its own separate sampler (or eventually samplers once its implemented for other samplers)

[AUTOMATIC1111]

do we have permission to copy and paste code from that repo

[space-nuko]

I filed an issue with the code owners here: https://github.com/wl-zhao/UniPC/issues/4

[wl-zhao]

Hi, thanks for your efforts! We are glad to see UniPC can also achieve good results in stable-diffusion-webui project. By the way, UniPC has already been integrated into diffusers, you can also refer to the doc if you have any questions.

[green-s]

Works great aside from a few things:

• Thresholding option is exposed despite only being supported for pixel-space models, as far as I can tell. Currently it just washes out the image.
• Not working with the dynamic thresholding extension: AttributeError: 'VanillaStableDiffusionSampler' object has no attribute 'model_wrap_cfg'. Not sure whose responsibility that is.
• Intermediate images are noisy. That's not the case with other samplers.
• Diffusers uses/recommends bh2 as the variant for >=10 steps but currently bh1 is used and bh2 isn't exposed.

[ghost]

https://huggingface.co/spaces/wl-zhao/unipc_sdm

This demo supports img2img. I could get impressive results with only 7 or 8 steps, with a well defined prompt

If I only need one or two steps for img2img then this would speed up making animations quite a lot. Wish this was implemented already

[vladmandic]

one request - handle borderline cases gracefully. currently unipc sampler throws an assertion if requested number of steps is below 3 (which is the default order and totally ). but a very common use case is to run through such scenarios (investigate intermediate steps, run with xyz grids on steps on one axies, etc.).

if number of steps is below acceptable limit, simpy use whatever is lowest possible. really, use assertions only for critical runtime errors, not for something that can and should be automatically handled.

Traceback (most recent call last):
  File "/home/vlado/dev/automatic/modules/call_queue.py", line 59, in f
    res = list(func(*args, **kwargs))
  File "/home/vlado/dev/automatic/modules/call_queue.py", line 38, in f
    res = func(*args, **kwargs)
  File "/home/vlado/dev/automatic/modules/txt2img.py", line 53, in txt2img
    processed = modules.scripts.scripts_txt2img.run(p, *args)
  File "/home/vlado/dev/automatic/modules/scripts.py", line 376, in run
    processed = script.run(p, *script_args)
  File "/home/vlado/dev/automatic/scripts/xyz_grid.py", line 617, in run
    processed = draw_xyz_grid(
  File "/home/vlado/dev/automatic/scripts/xyz_grid.py", line 292, in draw_xyz_grid
    process_cell(x, y, z, ix, iy, iz)
  File "/home/vlado/dev/automatic/scripts/xyz_grid.py", line 235, in process_cell
    processed: Processed = cell(x, y, z)
  File "/home/vlado/dev/automatic/scripts/xyz_grid.py", line 588, in cell
    res = process_images(pc)
  File "/home/vlado/dev/automatic/modules/processing.py", line 494, in process_images
    res = process_images_inner(p)
  File "/home/vlado/dev/automatic/modules/processing.py", line 640, in process_images_inner
    samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength, prompts=prompts)
  File "/home/vlado/dev/automatic/modules/processing.py", line 840, in sample
    samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
  File "/home/vlado/dev/automatic/modules/sd_samplers_compvis.py", line 185, in sample
    samples_ddim = self.launch_sampling(steps, lambda: self.sampler.sample(S=steps, conditioning=conditioning, batch_size=int(x.shape[0]), shape=x[0].shape, verbose=False, unconditional_guidance_scale=p.cfg_scale, unconditional_conditioning=unconditional_conditioning, x_T=x, eta=self.eta)[0])
  File "/home/vlado/dev/automatic/modules/sd_samplers_compvis.py", line 51, in launch_sampling
    return func()
  File "/home/vlado/dev/automatic/modules/sd_samplers_compvis.py", line 185, in <lambda>
    samples_ddim = self.launch_sampling(steps, lambda: self.sampler.sample(S=steps, conditioning=conditioning, batch_size=int(x.shape[0]), shape=x[0].shape, verbose=False, unconditional_guidance_scale=p.cfg_scale, unconditional_conditioning=unconditional_conditioning, x_T=x, eta=self.eta)[0])
  File "/home/vlado/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/vlado/dev/automatic/modules/models/diffusion/uni_pc/sampler.py", line 97, in sample
    x = uni_pc.sample(img, steps=S, skip_type=shared.opts.uni_pc_skip_type, method="multistep", order=shared.opts.uni_pc_order, lower_order_final=shared.opts.uni_pc_lower_order_final)
  File "/home/vlado/dev/automatic/modules/models/diffusion/uni_pc/uni_pc.py", line 751, in sample
    assert steps >= order, "UniPC order must be < sampling steps"
AssertionError: UniPC order must be < sampling steps

[vladmandic]

@space-nuko

I've just created a PR to bring UniPC sampler a bit closer to the rest of samplers used in WebUI #8589 And i've created a feature request if you can take a look? #8590

[Sakura-Luna]

I don't quite understand why UniPC becomes DDIM when using hires fix, it seems to introduce some noise?

Edit: Solved, the noise comes from the upscale step.