InvokeAI icon indicating copy to clipboard operation
InvokeAI copied to clipboard

Support FLUX LoRA models in kohya format with `lora_te1` layers (i.e. CLIP LoRA layers)

Open RyanJDick opened this issue 4 months ago • 0 comments

Summary

This PR add support for FLUX LoRA models in kohya format with lora_te1 layers (i.e. CLIP LoRA layers). Previously, only transformer LoRA layers were supported.

Example LoRA model in this format: https://huggingface.co/cocktailpeanut/optimus

Example

Prompt: optimus is playing tennis in a tennis court Seed: 0

Without LoRA: image

With LoRA: image

QA Instructions

I tested the following:

  • [x] The optimus LoRA (with CLIP layers) can be applied.
  • [x] FLUX LoRAs without CLIP layers still work
  • [x] Loading the optimus LoRA, but applying it to the transformer only produces a different result. I.e. verified that patching the CLIP layers is doing something. Ironically, the results seem better without applying the CLIP layers. The CLIP layers seem to pull in more background concepts. Regardless, it works.
  • [x] The optimus LoRA can be applied via the Linear UI, and the output matches results from manually constructing the workflow graph.
  • [x] FLUX LoRAs without CLIP layers still work via the Linear UI.

Checklist

  • [x] The PR has a short but descriptive title, suitable for a changelog
  • [x] Tests added / updated (if applicable)
  • [x] Documentation added / updated (if applicable)

RyanJDick avatar Sep 27 '24 14:09 RyanJDick