FlowMatch schedulers - closing the gap

[vladmandic]

As stated here, lets close the scheduler gap!

Problem statement:

• Most (if not all quality ones) new models are DiT based
• Implementation of DiT based models in Diffusers typically requires Flow-match type schedulers
• Currently only working scheduler is FlowMatchEulerDiscreteScheduler which creates a massive gap!

(just for completeness, there is also an older implementation of FlowMatchHeunDiscreteScheduler, but its not updated to support required set_timesteps or mu inputs, so it cannot be used with newer DiT models like Flux.1)

on the other hand, ComfyUI allows use of any regular scheduler with DiT and does not require separate duplicate -FlowMatch variant of the same scheduler (e.g. EulerDiscrete vs FlowMatchEulerDiscrete) with Flux.1.

reference: matrix of different schedulers used with Flux.1.Dev via ComfyUI:
https://civitai.com/articles/6582/flux1-dev-sampler-scheduler-comparison

cc: @yiyixuxu @sayakpaul @DN6 @asomoza

[yiyixuxu]

cc @hlky here too if interested!

[hlky]

I'd have to dig into FlowMatch a bit more in the community implementations, but it looks like there's little difference between EulerDiscrete and FlowMatchEulerDiscrete so FlowMatchEulerDiscrete could be integrated into EulerDiscrete controlled by some configuration option, this would avoid the need for separate duplicate variants and reduce maintenance burden - as there's already some divergence between schedulers, in set_timesteps for example, where some support sigmas/timesteps as input and others don't, support for final_sigmas_type, etc. Schedulers as a whole could probably do with an overhaul.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[vladmandic]

ping to remove stale.

[vladmandic]

any chance we get this one moving - given the popularity of flux and sd35 its one of the most frequent asks

[vladmandic]

@sayakpaul @yiyixuxu @asomoza as requested yesterday in a discussion, tagging this as a high-priority ask.

[sayakpaul]

This is definitely high-prio and is something @yiyixuxu is actively considering/looking into.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[nom]

Any progress on this @yiyixuxu @sayakpaul ? Still feels like a huge gap between Comfy and diffusers, especially with new samplers coming out like the now very popular RES4LYF (e.g. Qwen is basically unusable with Euler, way too smooth outputs)

[hlky]

The design surrounding samplers/schedulers in Diffusers has always been a pain point, spanning all the way back to 2022 where the separation of model and scheduler complicated the inclusion of 2nd order schedulers like DPM (https://github.com/huggingface/diffusers/issues/1308). More recently the rise of Flow Match models created the separate variants as per this issue, resulting in incompatability of various schedulers with Flux etc., some schedulers have since been "patched" with use_flow_sigmas/prediction_type="flow_prediction" which created further issues as not all Flow Match models use the same sigmas. Schedulers are also a blocker for async pipeline usage, the current recommendation for which is to provide each thread with its own scheduler instance, however this is not well documented.

My recommendation is to continue the work of #10146/https://github.com/huggingface/diffusers/discussions/11158 for a modular design with complete separation of sampling method (euler, etc.), schedule type (Beta, Flow), sigma schedule modifications (Beta, Exponential, Karras, adjustments as in works like Detail-Demon, etc.), ideally aim for everything to be stateless for example providing sigmas and the step index to the sampling method rather than storing those in the class, resolve any discrepancies such as how sigmas are passed from pipeline i.e. whether the final sigma is included, debloat as much as possible (or refactor as # Copied from could be used in more places as the code is the same, just written differently), provide methods for computing only pred original sample without actually making the sampling step as this is used in various works, and probably many other improvements can be made.

[bghira]

note: https://github.com/beinsezii/skrample provides a diffusers-compatible wrapper for a scheduler/sampler zoo.