kohya_ss
kohya_ss copied to clipboard
Published
20 hours ago •
bmaltais
bmaltais
After changing to SD3.flux branch - UnboundLocalError: local variable 'text_encoder_conds' referenced before assignment
hi, I am trying to train SDXL lora's again after trying the flux branch to test flux. Flux training works, but when I try to switch to SDXL again, I am getting an error. Previously had no issues. I removed the venv folder and did the setup.bat again, but issue remains.
I really don't want to have a second installation just for SDXL, so if someone can help figuring this out it will be greatly appreciated.
Here is the full command prompt log:
16:32:38-209857 INFO Kohya_ss GUI version: v24.2.0
16:32:38-767586 INFO Submodule initialized and updated.
16:32:38-773085 INFO nVidia toolkit detected
16:32:40-357364 INFO Torch 2.4.0+cu124
16:32:40-386917 INFO Torch backend: nVidia CUDA 12.4 cuDNN 90100
16:32:40-389917 INFO Torch detected GPU: NVIDIA GeForce RTX 4060 Ti VRAM 16379 Arch (8, 9) Cores 34
16:32:40-393918 INFO Python version is 3.10.11 (tags/v3.10.11:7d4cc5a, Apr 5 2023, 00:38:17) [MSC v.1929 64 bit
(AMD64)]
16:32:40-395918 INFO Verifying modules installation status from requirements_pytorch_windows.txt...
16:32:40-397919 INFO Verifying modules installation status from requirements_windows.txt...
16:32:40-399919 INFO Verifying modules installation status from requirements.txt...
16:32:48-650150 INFO headless: False
16:32:48-702660 INFO Using shell=True when running external commands...
Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.
16:33:13-689733 INFO Start training LoRA Standard ...
16:33:13-690733 INFO Validating lr scheduler arguments...
16:33:13-692733 INFO Validating optimizer arguments...
16:33:13-693734 INFO Validating J:\[Software 3]\Stable Diffusion\LoRA Training\zlogs existence and writability...
SUCCESS
16:33:13-694733 INFO Validating C:\temp\test1\model existence and writability... SUCCESS
16:33:13-696234 INFO Validating
C:/SD/stable-diffusion-webui/models/Stable-diffusion/ponyDiffusionV6XL_v6StartWithThisOne.safet
ensors existence... SUCCESS
16:33:13-697234 INFO Validating C:\temp\test1\Img existence... SUCCESS
16:33:13-698234 INFO Folder 1_n4114t3a: 1 repeats found
16:33:13-699734 INFO Folder 1_n4114t3a: 9 images found
16:33:13-700234 INFO Folder 1_n4114t3a: 9 * 1 = 9 steps
16:33:13-701235 INFO Regulatization factor: 1
16:33:13-702235 INFO Total steps: 9
16:33:13-703235 INFO Train batch size: 4
16:33:13-703735 INFO Gradient accumulation steps: 1
16:33:13-705235 INFO Epoch: 134
16:33:13-705735 INFO max_train_steps (9 / 4 / 1 * 134 * 1) = 302
16:33:13-707236 INFO stop_text_encoder_training = 0
16:33:13-707736 INFO lr_warmup_steps = 0
16:33:13-713737 INFO Saving training config to C:\temp\test1\model\Test1 - PonyXL V1_20240830-163313.json...
16:33:13-717237 INFO Executing command: C:\SD\Koya\kohya_ss\venv\Scripts\accelerate.EXE launch --dynamo_backend no
--dynamo_mode default --mixed_precision fp16 --num_processes 1 --num_machines 1
--num_cpu_threads_per_process 2 C:/SD/Koya/kohya_ss/sd-scripts/sdxl_train_network.py
--config_file C:\temp\test1\model/config_lora-20240830-163313.toml
C:\SD\Koya\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.
torch.utils._pytree._register_pytree_node(
C:\SD\Koya\kohya_ss\venv\lib\site-packages\xformers\ops\fmha\flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
@torch.library.impl_abstract("xformers_flash::flash_fwd")
C:\SD\Koya\kohya_ss\venv\lib\site-packages\xformers\ops\fmha\flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
@torch.library.impl_abstract("xformers_flash::flash_bwd")
C:\SD\Koya\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.
torch.utils._pytree._register_pytree_node(
C:\SD\Koya\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.
torch.utils._pytree._register_pytree_node(
2024-08-30 16:33:23 INFO Loading settings from train_util.py:4189
C:\temp\test1\model/config_lora-20240830-163313.toml...
INFO C:\temp\test1\model/config_lora-20240830-163313 train_util.py:4208
C:\SD\Koya\kohya_ss\venv\lib\site-packages\transformers\tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
warnings.warn(
2024-08-30 16:33:25 INFO Using DreamBooth method. train_network.py:281
INFO prepare images. train_util.py:1803
INFO get image size from name of cache files train_util.py:1741
100%|██████████████████████████████████████████████████████████████████████████████████| 9/9 [00:00<00:00, 6002.34it/s]
INFO set image size from cache files: 0/9 train_util.py:1748
INFO found directory C:\temp\test1\Img\1_n4114t3a contains 9 image files train_util.py:1750
WARNING No caption file found for 9 images. Training will continue without train_util.py:1781
captions for these images. If class token exists, it will be used. /
9枚の画像にキャプションファイルが見つかりませんでした。これらの画像につ
いてはキャプションなしで学習を続行します。class
tokenが存在する場合はそれを使います。
WARNING C:\temp\test1\Img\1_n4114t3a\1.jpg train_util.py:1788
WARNING C:\temp\test1\Img\1_n4114t3a\2.jpg train_util.py:1788
WARNING C:\temp\test1\Img\1_n4114t3a\3.jpg train_util.py:1788
WARNING C:\temp\test1\Img\1_n4114t3a\4.jpg train_util.py:1788
WARNING C:\temp\test1\Img\1_n4114t3a\5.jpg train_util.py:1788
WARNING C:\temp\test1\Img\1_n4114t3a\6.jpg... and 4 more train_util.py:1786
INFO 9 train images with repeating. train_util.py:1844
INFO 0 reg images. train_util.py:1847
WARNING no regularization images / 正則化画像が見つかりませんでした train_util.py:1852
INFO [Dataset 0] config_util.py:570
batch_size: 4
resolution: (1024, 1024)
enable_bucket: True
network_multiplier: 1.0
min_bucket_reso: 256
max_bucket_reso: 2048
bucket_reso_steps: 64
bucket_no_upscale: True
[Subset 0 of Dataset 0]
image_dir: "C:\temp\test1\Img\1_n4114t3a"
image_count: 9
num_repeats: 1
shuffle_caption: False
keep_tokens: 0
keep_tokens_separator:
caption_separator: ,
secondary_separator: None
enable_wildcard: False
caption_dropout_rate: 0.0
caption_dropout_every_n_epoches: 0
caption_tag_dropout_rate: 0.0
caption_prefix: None
caption_suffix: None
color_aug: False
flip_aug: False
face_crop_aug_range: None
random_crop: False
token_warmup_min: 1,
token_warmup_step: 0,
alpha_mask: False,
is_reg: False
class_tokens: n4114t3a
caption_extension: .caption
INFO [Dataset 0] config_util.py:576
INFO loading image sizes. train_util.py:876
100%|██████████████████████████████████████████████████████████████████████████████████| 9/9 [00:00<00:00, 5996.62it/s]
INFO make buckets train_util.py:882
WARNING min_bucket_reso and max_bucket_reso are ignored if bucket_no_upscale is train_util.py:899
set, because bucket reso is defined by image size automatically /
bucket_no_upscaleが指定された場合は、bucketの解像度は画像サイズから自動計
算されるため、min_bucket_resoとmax_bucket_resoは無視されます
INFO number of images (including repeats) / train_util.py:928
各bucketの画像枚数（繰り返し回数を含む）
INFO bucket 0: resolution (576, 1472), count: 1 train_util.py:933
INFO bucket 1: resolution (576, 1600), count: 1 train_util.py:933
INFO bucket 2: resolution (640, 1344), count: 1 train_util.py:933
INFO bucket 3: resolution (704, 832), count: 1 train_util.py:933
INFO bucket 4: resolution (768, 1152), count: 1 train_util.py:933
INFO bucket 5: resolution (768, 1280), count: 1 train_util.py:933
INFO bucket 6: resolution (832, 832), count: 1 train_util.py:933
INFO bucket 7: resolution (832, 1216), count: 1 train_util.py:933
INFO bucket 8: resolution (896, 896), count: 1 train_util.py:933
INFO mean ar error (without repeats): 0.019030180685547755 train_util.py:938
WARNING clip_skip will be unexpected / SDXL学習ではclip_skipは動作しません sdxl_train_util.py:352
INFO preparing accelerator train_network.py:335
C:\SD\Koya\kohya_ss\venv\lib\site-packages\accelerate\accelerator.py:488: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
self.scaler = torch.cuda.amp.GradScaler(**kwargs)
accelerator device: cuda
INFO loading model for process 0/1 sdxl_train_util.py:33
INFO load StableDiffusion checkpoint: sdxl_train_util.py:74
C:/SD/stable-diffusion-webui/models/Stable-diffusion/ponyDiffusionV6X
L_v6StartWithThisOne.safetensors
INFO building U-Net sdxl_model_util.py:198
INFO loading U-Net from checkpoint sdxl_model_util.py:202
2024-08-30 16:33:29 INFO U-Net: <All keys matched successfully> sdxl_model_util.py:208
INFO building text encoders sdxl_model_util.py:211
INFO loading text encoders from checkpoint sdxl_model_util.py:264
INFO text encoder 1: <All keys matched successfully> sdxl_model_util.py:278
2024-08-30 16:33:30 INFO text encoder 2: <All keys matched successfully> sdxl_model_util.py:282
INFO building VAE sdxl_model_util.py:285
INFO loading VAE from checkpoint sdxl_model_util.py:290
2024-08-30 16:33:31 INFO VAE: <All keys matched successfully> sdxl_model_util.py:293
INFO Enable xformers for U-Net train_util.py:3053
import network module: networks.lora
INFO [Dataset 0] train_util.py:2326
INFO caching latents with caching strategy. train_util.py:984
INFO checking cache validity... train_util.py:994
100%|████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:00<?, ?it/s]
INFO caching latents... train_util.py:1038
100%|████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:05<00:00, 1.67it/s]
2024-08-30 16:33:37 INFO create LoRA network. base dim (rank): 8, alpha: 8 lora.py:935
INFO neuron dropout: p=None, rank dropout: p=None, module dropout: p=None lora.py:936
INFO create LoRA for Text Encoder 1: lora.py:1027
INFO create LoRA for Text Encoder 2: lora.py:1027
INFO create LoRA for Text Encoder: 88 modules. lora.py:1035
INFO create LoRA for U-Net: 722 modules. lora.py:1043
INFO enable LoRA for U-Net: 722 modules lora.py:1089
prepare optimizer, data loader etc.
INFO use 8-bit AdamW optimizer | {} train_util.py:4342
running training / 学習開始
num train images * repeats / 学習画像の数×繰り返し回数: 9
num reg images / 正則化画像の数: 0
num batches per epoch / 1epochのバッチ数: 9
num epochs / epoch数: 34
batch size per device / バッチサイズ: 4
gradient accumulation steps / 勾配を合計するステップ数 = 1
total optimization steps / 学習ステップ数: 302
steps: 0%| | 0/302 [00:00<?, ?it/s]2024-08-30 16:33:44 INFO unet dtype: torch.float16, device: cuda:0 train_network.py:1030
INFO text_encoder dtype: torch.float16, device: cuda:0 train_network.py:1032
INFO text_encoder dtype: torch.float16, device: cuda:0 train_network.py:1032
epoch 1/34
INFO epoch is incremented. current_epoch: 0, epoch: 1 train_util.py:668
Traceback (most recent call last):
File "C:\SD\Koya\kohya_ss\sd-scripts\sdxl_train_network.py", line 210, in <module>
trainer.train(args)
File "C:\SD\Koya\kohya_ss\sd-scripts\train_network.py", line 1088, in train
text_encoder_conds is None
UnboundLocalError: local variable 'text_encoder_conds' referenced before assignment
steps: 0%| | 0/302 [00:00<?, ?it/s]
Traceback (most recent call last):
File "C:\Users\BloodRaven\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\BloodRaven\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\SD\Koya\kohya_ss\venv\Scripts\accelerate.EXE\__main__.py", line 7, in <module>
File "C:\SD\Koya\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 48, in main
args.func(args)
File "C:\SD\Koya\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1106, in launch_command
simple_launcher(args)
File "C:\SD\Koya\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 704, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['C:\\SD\\Koya\\kohya_ss\\venv\\Scripts\\python.exe', 'C:/SD/Koya/kohya_ss/sd-scripts/sdxl_train_network.py', '--config_file', 'C:\\temp\\test1\\model/config_lora-20240830-163313.toml']' returned non-zero exit status 1.
16:33:47-165420 INFO Training has ended.
