[Feature Request]: Multi-LoRA Composition

[catboxanon]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

Relatively new research that suggests a different method to handle how LoRA models are combined.

https://maszhongming.github.io/Multi-LoRA-Composition/ https://github.com/maszhongming/Multi-LoRA-Composition

https://github.com/AUTOMATIC1111/stable-diffusion-webui/assets/122327233/acf40c44-f63d-4640-a6c6-4f237344fa1a

Proposed workflow

New option in settings that modifies how LoRAs are combined.

• Merge (default)
• Switch
• Composite

Additional information

Merge is how the webui and most other implementations work. The Switch method is technically already possible with something like https://github.com/cheald/sd-webui-loractl. Composite seems to be the suggested new method and is the one the paper refers back to most often.

To explore incorporating all LoRAs at each timestep without merging weight matrices, we propose LoRA Composite. It involves calculating both unconditional and conditional score estimates for each LoRA individually at each step. By aggregating these scores, the technique ensures balanced guidance throughout the image generation process, facilitating the cohesive integration of all elements represented by different LoRAs.

https://github.com/maszhongming/Multi-LoRA-Composition/blob/eff93ef379b3842e8bcd049f2703852a99393e86/pipeline/pipeline_0.26.3.py#L1037