LyCORIS Dylora AttributeError apply_max_norm_regularization

[SmirkingKitsune]

I am having trouble running the LyCORIS Dylora adaption algorithm on an SD v1.5 checkpoint in Kohya_ss GUI v23.0.15. I am using this on a Windows 11 with RTX 3070. The environment uses Torch 2.1.2+cu118, CUDA 11.8, cuDNN 8905, triton 2.1.0, and Python 3.10.11.

When I try to use LyCORIS Dylora I get the following error:

2024-04-05 13:17:07|[LyCORIS]-INFO: Using rank adaptation algo: dylora
2024-04-05 13:17:07|[LyCORIS]-INFO: Use Dropout value: 0.0
2024-04-05 13:17:07|[LyCORIS]-INFO: Create LyCORIS Module
2024-04-05 13:17:08|[LyCORIS]-INFO: create LyCORIS for Text Encoder: 72 modules.
2024-04-05 13:17:08|[LyCORIS]-INFO: Create LyCORIS Module
2024-04-05 13:17:09|[LyCORIS]-INFO: create LyCORIS for U-Net: 390 modules.
2024-04-05 13:17:09|[LyCORIS]-INFO: module type table: {'DyLoraModule': 354, 'NormModule': 108}
Traceback (most recent call last):
  File "Z:\ProgramData\kohya_ss\sd-scripts\train_network.py", line 1058, in <module>
    trainer.train(args)
  File "Z:\ProgramData\kohya_ss\sd-scripts\train_network.py", line 300, in train
    network = network_module.create_network(
  File "Z:\ProgramData\kohya_ss\venv\lib\site-packages\lycoris\kohya\__init__.py", line 148, in create_network
    delattr(type(network), "apply_max_norm_regularization")
AttributeError: apply_max_norm_regularization
Traceback (most recent call last):
  File "C:\Users\sd\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\sd\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "Z:\ProgramData\kohya_ss\venv\Scripts\accelerate.exe\__main__.py", line 7, in <module>
  File "Z:\ProgramData\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main
    args.func(args)
  File "Z:\ProgramData\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1017, in launch_command
    simple_launcher(args)
  File "Z:\ProgramData\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 637, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['Z:\\ProgramData\\kohya_ss\\venv\\Scripts\\python.exe', 'Z:\\ProgramData\\kohya_ss/sd-scripts/train_network.py', '--max_grad_norm=0', '--bucket_reso_steps=1', '--cache_latents', '--cache_latents_to_disk', '--caption_extension=.txt', '--enable_bucket', '--min_bucket_reso=256', '--max_bucket_reso=2048', '--flip_aug', '--learning_rate=4e-07', '--logging_dir=Z:\\Stable_Diffusion_Training\\SD_Scripts\\Dev\\log', '--lr_scheduler=constant_with_warmup', '--lr_scheduler_num_cycles=5', '--lr_warmup_steps=13012', '--max_data_loader_n_workers=0', '--resolution=512,512', '--max_token_length=225', '--max_train_steps=162650', '--min_snr_gamma=10', '--mixed_precision=bf16', '--network_alpha=64', '--network_args', 'preset=full', 'conv_dim=64', 'conv_alpha=64', 'use_tucker=False', 'block_size=1', 'rank_dropout=0', 'module_dropout=0', 'algo=dylora', 'train_norm=True', '--network_dim=64', '--network_module=lycoris.kohya', '--network_train_unet_only', '--noise_offset=0.0357', '--optimizer_args', 'scale_parameter=False', 'relative_step=False', 'warmup_init=False', '--optimizer_type=Adafactor', '--output_dir=Z:\\Stable_Diffusion_Training\\SD_Scripts\\Dev\\model', '--output_name=DevV3.1', '--pretrained_model_name_or_path=Z:/ProgramData/Stable-Diffusion-webui/webui/models/Stable-diffusion/CustomMix_v3.safetensors', '--save_every_n_epochs=1', '--save_model_as=safetensors', '--save_precision=fp16', '--seed=1234', '--shuffle_caption', '--train_batch_size=1', '--training_comment=V3.1', '--train_data_dir=Z:\\Stable_Diffusion_Training\\SD_Scripts\\Dev\\img', '--unet_lr=4e-07', '--xformers', '--sample_sampler=euler_a', '--sample_prompts=Z:\\Stable_Diffusion_Training\\SD_Scripts\\Dev\\model\\sample\\prompt.txt', '--sample_every_n_epochs=1']' returned non-zero exit status 1.

I have tried reinstalling the venv but got the same result. For both attempts, I have used bitsandbytes 0.43.0 or bitsandbytes-windows. I tried running Kohya_ss GUI v23.0.15 with Kohya_ss v0.8.4 instead of Kohya_ss v0.8.5 and got the same problem. I was previously using Kohya_ss v0.8.4 on Kohya_ss GUI v22.6.2 and that worked fine, so I don't think it is a Kohya_ss problem.

I asked about this on Kohya GUI's repo and @bmaltais helped me troubleshoot. @bmaltais suggested that I come here to ask about this issue and also gave me a temporary workaround for __init__.py:

if algo == "dylora":
    # Attempt to remove the unsupported feature from the `network` class
    try:
        delattr(type(network), "apply_max_norm_regularization")
    except AttributeError:
        # If the attribute doesn't exist, do nothing
        pass

This workaround seems to work, but since I don't know the root cause of the attribute error, I am unsure if this might present further issues down the line.

Since I knew that Kohya_ss GUI v22.6.2 worked fine, I tried using the lycoris_lora module that Kohya_ss GUI v22.6.2 used in Kohya_ss GUI v23.0.15. Kohya_ss GUI v23.0.15 uses lycoris_lora 2.2.0.post3, and Kohya_ss GUI v22.6.2 uses lycoris_lora 2.0.2. So I replaced the lycoris_lora 2.2.0.post3 module with the lycoris_lora 2.0.2 module in Kohya_ss GUI v23.0.15 and did not encounter the attribute error.

[SmirkingKitsune]

I checked to see if this attribute error was present in lycoris_lora 2.1.0, and I got the aforementioned AttributeError: apply_max_norm_regularization error.

[KohakuBlueleaf]

@RookHyena Some updates here: This issue will eventually be fixed but not recent dylora/IA3/lilora/glora are all outdated and I decide to rewrite them (to match my new API format)

You may need to wait until I finish this whole reconstruction

[SmirkingKitsune]

Okay. It's good to know that this is a planned feature.

[KohakuBlueleaf]

@SmirkingKitsune These kind of things is totally ressolved in 3.0.0.dev6 (latest dev) you can try to install it pip install -U --pre lycoris-lora