Request: FLUX Model (FluxFillPipeline, FluxPriorReduxPipeline) from diffusers.from_pretrained to ComfyUI Node Loading Guide

[muxuezzz]

trafficstars

Your question

Problem Description Currently, when calling methods like FluxFillPipeline.from_pretrained() or FluxPriorReduxPipeline.from_pretrained() in the diffusers library to load FLUX models, the entire Hugging Face repository is downloaded by default. This includes all relevant component weights and configuration files (Transformer, VAE, text encoders, scheduler, etc.).

While this method is convenient and quick, it presents the following limitations for ComfyUI, a platform centered around modular, node-based workflows:

Unnecessary Full Repository Download: When users only need the core model file (e.g., flux1-dev.safetensors) and wish to combine it with other local or custom components, downloading the entire repository leads to wasted storage space and increased download times.

Limited Workflow Dynamism: The from_pretrained method loads a predefined, complete pipeline. This restricts the flexibility of ComfyUI node developers to dynamically replace or combine different versions of VAEs, text encoders, or schedulers when building workflows. ComfyUI's core strength lies in its fine-grained component control.

Desired Solution We hope that ComfyUI can provide official code guidance or best practices to help node developers convert model calls based on diffusers.FluxFillPipeline and diffusers.FluxPriorReduxPipeline to use existing or potentially future core ComfyUI nodes (e.g., LoadUnet, Load CLIP, Load VAE, DiffusersSchedulerLoader, etc.).

Specifically, the guidance should focus on:

How to load only the core flux1-dev.safetensors file of the FLUX model as the main Transformer component.

How to identify and load compatible VAEs (e.g., AutoencoderKL) and text encoders (e.g., CLIPTextModel and T5EncoderModel) from .safetensors files or via other nodes.

How to instantiate and configure the necessary scheduler for the FLUX pipeline (e.g., FlowMatchEulerDiscreteScheduler).

Ultimately, how to assemble a fully functional and efficient FLUX generation workflow through ComfyUI's node connections.

Why This Is Important Optimize Resource Utilization: Avoid unnecessary full repository downloads, reducing storage footprint and network bandwidth consumption.

Enhance Flexibility and Customization: Allow users and node developers to freely combine different versions of FLUX components (e.g., using different VAEs or text encoders) based on their needs, leading to more flexible and experimental workflows.

Align with ComfyUI's Design Philosophy: Further strengthen ComfyUI's advantages in modularity and composability.

Improve Developer Experience: Provide clear, actionable guidance for node developers who wish to integrate FLUX pipelines from diffusers into ComfyUI's framework.

Current Understanding and Challenges We understand that the flux1-dev.safetensors file primarily contains the core Transformer component weights of the FLUX model. In diffusers, FluxTransformer2DModel.from_single_file() can be used to load this core file. However, a complete FluxPipeline still requires auxiliary components such as VAEs, text encoders, and schedulers. Currently, these auxiliary components in diffusers typically still need to be downloaded via from_pretrained from their respective Hugging Face repositories (e.g., openai/clip-vit-large-patch14 for CLIP, google/umt5-xxl for T5, madebyollin/sdxl-vae-fp16-fix for VAE), unless the user has local .safetensors files for these components.

We look forward to ComfyUI providing specific guidance for FluxFillPipeline and FluxPriorReduxPipeline to help community developers implement this component-level loading and assembly more elegantly within the ComfyUI framework.

Thank you for your consideration!

Logs

Other

No response