[Bug]: Cannot load LoRA models created with Dreambooth extension or Colab dynamically

[Danamir]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What happened?

Using the "Show extra network" I can load LoRA files downloaded online, but when creating a LoRA file locally using the Dreambooth extension on Windows, or in a Colab, the file created cannot be loaded dynamically, provoking this error :

activating extra network lora with arguments [<modules.extra_networks.ExtraNetworkParams object at 0x000002054665FB20>]: TypeError
Traceback (most recent call last):
  File ".\stable-diffusion-webui\modules\extra_networks.py", line 75, in activate
    extra_network.activate(p, extra_network_args)
  File ".\stable-diffusion-webui\extensions-builtin\Lora\extra_networks_lora.py", line 17, in activate
    lora.load_loras(names, multipliers)
  File ".\stable-diffusion-webui\extensions-builtin\Lora\lora.py", line 151, in load_loras
    lora = load_lora(name, lora_on_disk.filename)
  File ".\stable-diffusion-webui\extensions-builtin\Lora\lora.py", line 83, in load_lora
    sd = sd_models.read_state_dict(filename)
  File ".\stable-diffusion-webui\modules\sd_models.py", line 227, in read_state_dict
    sd = get_state_dict_from_checkpoint(pl_sd)
  File ".\stable-diffusion-webui\modules\sd_models.py", line 198, in get_state_dict_from_checkpoint
    pl_sd = pl_sd.pop("state_dict", pl_sd)
TypeError: pop expected at most 1 argument, got 2

Having inspected the expected input in a Python debugger, it seems the generated LoRA files contains a list object ; where the ones downloaded online contains a dict.

I'm not sure if I'm missing a step to convert the training LoRA output to something useful, or if it's supposed to be loadable.

Steps to reproduce the problem

1. Use the Dreambooth extension or https://colab.research.google.com/drive/1iSFDpRBKEWr2HLlz243rbym3J2X95kcy to train a LoRA file
2. Once the training is completed, try to load the LoRA file using the "Show extra network" button
3. An error is produced in the log output, and the LoRA is not dynamically merged to the current checkpoint

What should have happened?

The LoRA file from training should update on-the-fly the current model to alter the generated outputs, as the downloaded ones from civitai for example.

Commit where the problem happens

602a1864b05075ca4283986e6f5c7d5bce864e11

What platforms do you use to access UI ?

Windows

What browsers do you use to access the UI ?

Mozilla Firefox, Google Chrome

Command Line Arguments

--xformers

Additional information, context and logs

No response

[Danamir]

When inspecting the pl_sd variable content, the final data is the same, but the downloaded one is in a dict with meaningful keys, while the trained one is stored in a list. The code in sd_models.py expects a dict later : for k, v in pl_sd.items(): .

Here is the view from the Python debugger when loading a downloaded LoRA file :

And a LoRA file generated with training :

[karolisr]

I think this may be more easily addressed by the Dreambooth UI maintainer. It seems they may not be saving the lora model in the way that is compatible. I may be wrong; I know very little about the formats here. But looking at the Dreambooth UI code:

When saving, Dreambooth UI calls save_lora_weight function here: https://github.com/d8ahazard/sd_dreambooth_extension/blob/9f4d931a319056c537d24669cb950d146d1537b0/dreambooth/train_dreambooth.py#L678

save_lora_weight function is implemented here: https://github.com/d8ahazard/sd_dreambooth_extension/blob/9f4d931a319056c537d24669cb950d146d1537b0/lora_diffusion/lora.py#L197

It dumps everything into a list. However, there is a function in Dreambooth UI's lora.py file called save_safeloras which produces a dictionary: https://github.com/d8ahazard/sd_dreambooth_extension/blob/9f4d931a319056c537d24669cb950d146d1537b0/lora_diffusion/lora.py#L226

It seems that this may be the format expected by AUTOMATIC1111 UI.

[Prsaro]

same problem

[remybonnav]

same problem using webui on google colab from Thelastben. Any fix???

[VeryEvilHumna]

Still no fix. Is here some other ways to use dreambooth loras?

[Vynavo]

Is anyone trying to figure this out or did anyone find a solution? The bug does still exist.

@kohya-ss do you know about this?

[leandro-gracia-gil]

Maybe it would be worth reporting this also as an issue to the Dreambooth extension repo, linking back here.

[SilentD1]

I use kohya to extraxt the lora from the ckpt file, but yeah would be nicer to just have it done automatically

[reserved-word]

Lora also may be extracted using the sd-webui-supermerger extension. But it worked for me only for 1.5 model, I got an error on the 2.1 model

[Gushousekai195]

Lora also may be extracted using the sd-webui-supermerger extension. But it worked for me only for 1.5 model, I got an error on the 2.1 model

How is that done?

[reserved-word]

@Gushousekai195 In A1111 ui with installed extension:

1. SuperMerger tab > LoRA tab
2. Checkpoint A = your trained model that contains merged lora (ckpt/safetensors)
3. Checkpoint B = model that you used as base for training
4. alpha = 1, beta = 1 (if you want to get the "clean", not mixed LoRa)
5. Remake dimension = Lora rank that you used (4 in Dreambooth extension by default)
6. filename = name of result lora file
7. Press "Make LoRA (alpha * A - beta * B)" and wait few minutes

[cerarslan]

Lora also may be extracted using the sd-webui-supermerger extension. But it worked for me only for 1.5 model, I got an error on the 2.1 model

Hello mate, can you use 2.1 based lora with extra network?

[walzuu]

I changed the stable-diffusion-webui\extensions\sd_dreambooth_extension\lora_diffusion\lora.py method save_lora_weight likes this:

def save_lora_weight(
        model,
        path="./lora.pt",
        target_replace_module=None,
        save_safetensors: bool = False,
        d_type: dtype = torch.float32
):
    if target_replace_module is None:
        target_replace_module = DEFAULT_TARGET_REPLACE
    weights = {}
    idx = 0

    for _up, _down in extract_lora_ups_down(
            model, target_replace_module=target_replace_module
    ):
        weights["up"+str(idx)]=_up.weight.to("cpu", dtype=d_type)
        weights["down"+str(idx)]=_down.weight.to("cpu", dtype=d_type)
        idx += 1

    if save_safetensors:
        path = path.replace(".pt", ".safetensors")
        save_safeloras(weights, path)
    else:
        torch.save(weights, path)

and it seemed to work. Not sure if the dictionary keys should have some proper values.

[JBrainstorming]

If anyone know how to fix this? I'm having trouble adding a useful extension named 'Kohya--ss Additional Networks script' model into my stable diffusion webui. Really want to know how to activate this extension without clicking the button:

[JBrainstorming]

If anyone know how to fix this? I'm having trouble adding a useful extension named 'Kohya--ss Additional Networks script' model into my stable diffusion webui. Really want to know how to activate this extension without clicking the button:

Since it doesn't react when clicked, it would be useless to "Apply and restart UI".

[SebaSpeed]

In my case, after trying to solve it in a thousand different ways, I managed to solve it updating everything correctly and verifying the versions very well. Maybe it was only the problem of using "Lora Extended" since now the system warns me that a module is missing, I have to try to install it. Another important fact is that you cannot place the files ".pt" directly in "Models/Lora", these do not work for that case; When you generate the Loras with the correct options that folder is automatically occupied with the generated files ".safetensors". Many of those who had this problem were for these various reasons, sometimes it is convenient to start from scratch and verify everything very well so as not to waste our time and that of the developers. Just my 2 cents ;)