More Samplers/Schedulers, e.g. Euler

[dblunk88]

Thank you guys for throwing in the documentation. Segregating this from the other Issue.

Is there any possibility to include more of the k-diffusion samplers? https://github.com/crowsonkb/k-diffusion/blob/master/k_diffusion/sampling.py

Heuler for example would be nice

[ElleSimpsonEdin]

Seconded!

[mja]

Looking at the code and the equations referenced, sample_euler in k_diffusion is the same or very similar to KarrasVeScheduler in diffusers. So that would be a good starting point for implementing the other samplers (sample_huen, sample_dpm_2) that take the same parameters (s_churn, s_tmin, s_tmax, s_noise).

[exo-pla-net]

TLDR: Now that we have optimized VRAM usage, I'd argue that the next feature to expand is samplers. Devs are attached to certain samplers, and their absence here is tempting devs to develop outside of huggingface. Given that a lot of devs are attached to k_euler_a in particular, I think adding it to Hugging Face should be a high priority.

---

Below is my overview of where we are with samplers, and what's important.

Currently, our diffusers project has these samplers available: DDIM, LMS, or PNDM.

It appears the gameplan is to make separate files for each sampler we implement.

What other samplers exist? Where can you try them out and see how they're implemented? I'd suggest the hlky Colab here.

Their samplers: "DDIM", "PLMS", 'k_dpm_2_a', 'k_dpm_2', 'k_euler_a', 'k_euler', 'k_heun', 'k_lms'. (Our PNDM ~= their PLMS. Our LMS ~= their k_lms.)

But wait, you might ask, do samplers even matter? Is choosing the correct sampler important for making a great image? I have looked at several comparisons between samplers. Here are three of the best: one , two, and three. My conclusions:

• Given low CFG and >=25 steps, all samplers are fairly interchangeable.
• Accommodating high CFG, however, is very important for defining the look of an image. -- Low CFG = washed out, greater intricacy. -- High CFG == more defined, higher saturation.
• Given high CFG, some schedulers require 50 or more steps to get rid of a seed's rainbow artifacts.
• Steps are different speeds in different samplers. Some samplers are much quicker.

So which is the best sampler? I think the following reasoning is a solid foundation for making that choice:

• For any given prompt+CFG, there exists a "steady state" number of inference steps, where an additional step is equally likely to make the image better or worse.
• Steady state for many excellent images is achieved in merely 25 steps.
• Some samplers, however, do not reliably eliminate the rainbow noise in 25 steps.

And thus, the best scheduler is the one that resolves high-CFG rainbow noise in 25 inference steps the fastest. For that purpose, these samplers are all equivalent: DDIM, k_euler, k_euler_a. Using any of those, you'll be able to explore seed+CFG space the fastest, and thereby create a great image the fastest.

However, I and other devs have developed a superstition about k_euler_a. There's just something to it. k_euler_a is a contrarian sampler that refuses to resolve, no matter how many steps you throw at it. It seems to often spit out higher quality images than other samplers. (I think k_euler_a is possibly just chaotic, and you'd find an equal number of great images in equal time, simply by exploring different seeds with another sampler, instead of pounding on the same seed with the chaotic k_euler_a. But then again, maybe not -- maybe k_euler_a truly is special. Can anybody math me and say something definitive?)

We want developers to feel like Hugging Face's diffusers is all they need to make state-of-the-art apps. Whether or not it's merely superstition that makes devs want to use other samplers, we should try to include those samplers, because great devs are not immune to superstition, and we want them building with Hugging Face.

(Plus, you never know, k_euler_a might add some extra spice to our incantations /~*~~ )

[Inkorak]

Yeah, k_euler_a has some popularity. It is really worth implementing, at least as a selling point.

[sidthekidder]

Thanks for the context above @exo-pla-net! I'm planning to try and implement the sample_euler_ancestral sampler in a new scheduler. I read the schedulers/scheduling_karras_ve.py implementation that's similar to sample_euler so I'll structure my code similarly.

A question - since the Algorithm 2 euler implementation is also spread across the pipelines/stochatic_karras_ve/pipeline_stochastic_karras_ve.py file, does that mean sample_euler_ancestral would require a new pipeline as well? Or should I try to modify the existing pipeline_stochastic_karras_ve.py, since it's similar apart from adding ancestral sampling?

[Arkitecc]

I'm very much looking forward to this, if anyone is able to get this down definitely tag me.

[Suhail]

it would be useful to implement as many of the samplers that DreamStudio does: https://github.com/Stability-AI/stability-sdk

Users are becoming used to using them.

[patrickvonplaten]

@patil-suraj @anton-l - plans/ideas on how to move forward here?

[crowsonkb]

k-diffusion's Euler and Euler Ancestral samplers are just the VE versions of DDIM and the original DDPM sampling method though, they should actually produce the same outputs as their VP counterparts if you convert a VP diffusion process to VE...

[patrickvonplaten]

@anton-l any updates here?

[patil-suraj]

There's a PR open here #944 should be merged soon with Euler and Euler A scheduler

[dblunk88]

Awesome work!! That is amazing. Should I leave this open until all of them are migrated or should I close the issue?

[dblunk88]

I'm very much looking forward to this, if anyone is able to get this down definitely tag me.

@Arkitecc

[patrickvonplaten]

Think we can leave it open in case someone is interested in adding even more schedulers :-)

[patrickvonplaten]

I think we pretty much have all existing schedulers now in diffusers - are we missing one still?

[Suhail]

Do you guys support these?

Is k_dpm_2 DPMSolverMultistepScheduler?

[keturn]

I think

• k_dpm_2: DPMSolverMultistepScheduler(algorithm_type='dpmsolver')
• k_dpmpp_2: DPMSolverMultistepScheduler(algorithm_type='dpmsolver++') [default]

not sure about the _ancestral variants, whether that corresponds to one of the many configuration options for DPMSolverMultistepScheduler, or if that's a crowsonkb special addition that wasn't in Cheng Lu's implementation for diffusers.

[patrickvonplaten]

Actually sorry I think there is some confusion because the names are so similar :sweat_smile:

Here the conversion k-diffusion<>diffusers:

• sample_dpm_2: https://huggingface.co/docs/diffusers/api/schedulers/dpm_discrete
• sample_dpm_2_ancestral: https://huggingface.co/docs/diffusers/api/schedulers/dpm_discrete_ancestral
• k_heun: https://huggingface.co/docs/diffusers/api/schedulers/heun
• sample_dpmpp_sde: none - diffusers does not have this one as it's an unofficial one that Katherine created I think :-) Happy to add it though
• sample_dpmpp_2m: https://huggingface.co/docs/diffusers/api/schedulers/multistep_dpm_solver
• sample_dpmpp_sde_ancestral: This is the DPM++ single-step (2s) scheduler, but in the ancestral version. We have the official single-step deterministic version here: https://huggingface.co/docs/diffusers/api/schedulers/singlestep_dpm_solver

Hope this helps a bit :-)

[patrickvonplaten]

Here is a doc explaining how to use them: https://huggingface.co/docs/diffusers/using-diffusers/schedulers

[Suhail]

Very helpful - thank you

[usamaehsan]

Do you guys support DPM adaptive? I am only getting good results with this scheduler

[patrickvonplaten]

Hey @usamaehsan,

Can you send a link to DPM adaptive? Where is it from?

[usamaehsan]

Hey @usamaehsan,

Can you send a link to DPM adaptive? Where is it from?

It's from k-diffusion: https://github.com/crowsonkb/k-diffusion/blob/686dbad0f39640ea25c8a8c6a6e56bb40eacefa2/k_diffusion/sampling.py

I am using it with automatic 1111, stable diffusion web ui

[pete-finesse]

This scheduler k_dpmpp_2s_a (or k_dpmpp_2 generally) is mentioned in this thread before but I can't figure out how it maps to the diffusers schedulers. Does it exist and I'm just missing something? This is the method for reference