Using ControlNet with LoRA

[asadm]

What would be the right way to control a LoRA patched model using ControlNet?

Also, I wonder if we also need to patch controlnet model using something like this (Discussion).

[haofanwang]

This two repos may help. First, you can convert controlnet into diffusers supported format, then just treat it as a regular SD model, and further add LoRA layers. @asadm

ControlNet-for-Diffusers

Easy-Lora-Handbook

[cian0]

but would this work with @cloneofsimo 's approach of patching pipe? The lora there seems to have been trained using kohya's approach

[zengqixun12]

This two repos may help. First, you can convert controlnet into diffusers supported format, then just treat it as a regular SD model, and further add LoRA layers. @asadm

ControlNet-for-Diffusers

Easy-Lora-Handbook

I tried this method, but fialed on the second step: ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ /usr/local/mnn/ENV/bin/lora_add:8 in │ │ │ │ 5 from lora_diffusion.cli_lora_add import main │ │ 6 if name == 'main': │ │ 7 │ sys.argv[0] = re.sub(r'(-script.pyw|.exe)?$', '', sys.argv[0]) │ │ ❱ 8 │ sys.exit(main()) │ │ 9 │ │ │ │ /usr/local/mnn/ENV/lib/python3.8/site-packages/lora_diffusion/cli_lora_add.py:187 in main │ │ │ │ 184 │ │ 185 │ │ 186 def main(): │ │ ❱ 187 │ fire.Fire(add) │ │ 188 │ │ │ │ /usr/local/mnn/ENV/lib/python3.8/site-packages/fire/core.py:141 in Fire │ │ │ │ 138 │ context.update(caller_globals) │ │ 139 │ context.update(caller_locals) │ │ 140 │ │ ❱ 141 component_trace = _Fire(component, args, parsed_flag_args, context, name) │ │ 142 │ │ 143 if component_trace.HasError(): │ │ 144 │ _DisplayError(component_trace) │ │ │ │ /usr/local/mnn/ENV/lib/python3.8/site-packages/fire/core.py:475 in _Fire │ │ │ │ 472 │ is_class = inspect.isclass(component) │ │ 473 │ │ │ 474 │ try: │ │ ❱ 475 │ │ component, remaining_args = _CallAndUpdateTrace( │ │ 476 │ │ │ component, │ │ 477 │ │ │ remaining_args, │ │ 478 │ │ │ component_trace, │ │ │ │ /usr/local/mnn/ENV/lib/python3.8/site-packages/fire/core.py:691 in _CallAndUpdateTrace │ │ │ │ 688 │ loop = asyncio.get_event_loop() │ │ 689 │ component = loop.run_until_complete(fn(*varargs, **kwargs)) │ │ 690 else: │ │ ❱ 691 │ component = fn(*varargs, **kwargs) │ │ 692 │ │ 693 if treatment == 'class': │ │ 694 │ action = trace.INSTANTIATED_CLASS │ │ │ │ /usr/local/mnn/ENV/lib/python3.8/site-packages/lora_diffusion/cli_lora_add.py:142 in add │ │ │ │ 139 │ │ │ path_1, │ │ 140 │ │ ).to("cpu") │ │ 141 │ │ │ │ ❱ 142 │ │ tok_dict = patch_pipe(loaded_pipeline, path_2, patch_ti=False) │ │ 143 │ │ │ │ 144 │ │ collapse_lora(loaded_pipeline.unet, alpha_1) │ │ 145 │ │ collapse_lora(loaded_pipeline.text_encoder, alpha_1) │ │ │ │ /usr/local/mnn/ENV/lib/python3.8/site-packages/lora_diffusion/lora.py:1012 in patch_pipe │ │ │ │ 1009 │ │ │ 1010 │ elif maybe_unet_path.endswith(".safetensors"): │ │ 1011 │ │ safeloras = safe_open(maybe_unet_path, framework="pt", device="cpu") │ │ ❱ 1012 │ │ monkeypatch_or_replace_safeloras(pipe, safeloras) │ │ 1013 │ │ tok_dict = parse_safeloras_embeds(safeloras) │ │ 1014 │ │ if patch_ti: │ │ 1015 │ │ │ apply_learned_embed_in_clip( │ │ │ │ /usr/local/mnn/ENV/lib/python3.8/site-packages/lora_diffusion/lora.py:800 in │ │ monkeypatch_or_replace_safeloras │ │ │ │ 797 │ │ 798 │ │ 799 def monkeypatch_or_replace_safeloras(models, safeloras): │ │ ❱ 800 │ loras = parse_safeloras(safeloras) │ │ 801 │ │ │ 802 │ for name, (lora, ranks, target) in loras.items(): │ │ 803 │ │ model = getattr(models, name, None) │ │ │ │ /usr/local/mnn/ENV/lib/python3.8/site-packages/lora_diffusion/lora.py:562 in parse_safeloras │ │ │ │ 559 │ keys.sort(key=get_name) │ │ 560 │ │ │ 561 │ for name, module_keys in groupby(keys, get_name): │ │ ❱ 562 │ │ info = metadata.get(name) │ │ 563 │ │ │ │ 564 │ │ if not info: │ │ 565 │ │ │ raise ValueError( │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ AttributeError: 'NoneType' object has no attribute 'get'

[JamesSand]

hi I have the same problem with you, in merging lora and controlnet, is there any solution for this?

[cian0]

The official diffusers implementation of controlnet already works with simo's lora

[JamesSand]

The official diffusers implementation of controlnet already works with simo's lora

hi is there any example for this?

[ShashwatNigam99]

The official diffusers implementation of controlnet already works with simo's lora

Could you please elaborate on this? How can I make this work?