[Feature Request]: Multi-LoRA Composition

[catboxanon]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

Relatively new research that suggests a different method to handle how LoRA models are combined.

https://maszhongming.github.io/Multi-LoRA-Composition/ https://github.com/maszhongming/Multi-LoRA-Composition

https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/122327233/acf40c44-f63d-4640-a6c6-4f237344fa1a

Proposed workflow

New option in settings that modifies how LoRAs are combined.

• Merge (default)
• Switch
• Composite

Additional information

Merge is how the webui and most other implementations work. The Switch method is technically already possible with something like https://github.com/cheald/sd-webui-loractl (only in A1111 currently). Composite seems to be the suggested new method and is the one the paper refers back to most often.

To explore incorporating all LoRAs at each timestep without merging weight matrices, we propose LoRA Composite. It involves calculating both unconditional and conditional score estimates for each LoRA individually at each step. By aggregating these scores, the technique ensures balanced guidance throughout the image generation process, facilitating the cohesive integration of all elements represented by different LoRAs.

https://github.com/maszhongming/Multi-LoRA-Composition/blob/eff93ef379b3842e8bcd049f2703852a99393e86/pipeline/pipeline_0.26.3.py#L1037

I've opened a relevant issue upstream but since LoRAs are handled entirely different here I figured I'd open an issue here as well.

[altoiddealer]

I was reading through this carefully, and it seems like this isn't compatible with traditional LORA models? It seems more like their dataset is comprised of 22 specialized loras that work together...

[catboxanon]

Those 22 are just the LoRAs selected for their testbed. The paper mentions the LoRAs used are existing public ones.

[the ComposLoRA] testbed builds upon a collection of public LoRAs, collected from civitai.com, which are extensively shared and recognized as essential plug-in modules in this field.

[catboxanon]

Tried implementing this, results look worse though, so I'm going to close this as I don't see a point in supporting it. See https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/15037#issuecomment-2002655419 for an in-depth explantation.