Model hangs on caching latents

[starmancy42]

I've gone about re-installing this a few times now, but for some reason whenever I try to train the model, it just gets stuck on caching latents and it never updates or proceeds through with it:

Folder 100_TestChar: 3100 steps max_train_steps = 1550 stop_text_encoder_training = 0 lr_warmup_steps = 0 accelerate launch --num_cpu_threads_per_process=2 "train_db.py" --v2 --pretrained_model_name_or_path="stabilityai/stable-diffusion-2-base" --train_data_dir="C:/Users/DW-HEPHAEUSTUS-2563/Desktop/Kohya/Char Models/TestLORA/Image" --resolution=768,768 --output_dir="C:/Users/DW-HEPHAEUSTUS-2563/Desktop/Kohya/Char Models/TestLORA/Output" --logging_dir="C:/Users/DW-HEPHAEUSTUS-2563/Desktop/Kohya/Char Models/TestLORA/log" --save_model_as=safetensors --output_name="TestChar" --max_data_loader_n_workers="1" --learning_rate="0.0001" --lr_scheduler="constant" --train_batch_size="2" --max_train_steps="1550" --save_every_n_epochs="1" --mixed_precision="bf16" --save_precision="bf16" --seed="1234" --caption_extension=".txt" --cache_latents --max_data_loader_n_workers="1" --clip_skip=2 --bucket_reso_steps=64 --xformers --use_8bit_adam --bucket_no_upscale v2 with clip_skip will be unexpected / v2でclip_skipを使用することは想定されていません prepare tokenizer prepare train images. found directory 100_TestChar contains 31 image files 3100 train images with repeating. loading image sizes. 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 31/31 [00:00<00:00, 3441.41it/s] prepare dataset prepare accelerator Using accelerator 0.15.0 or above. load Diffusers pretrained models text_encoder\model.safetensors not found Fetching 16 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:00<00:00, 15978.30it/s] C:\Users\DW-HEPHAEUSTUS-2563\Desktop\Kohya\kohya_ss\venv\lib\site-packages\transformers\models\clip\feature_extraction_clip.py:28: FutureWarning: The class CLIPFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use CLIPImageProcessor instead. warnings.warn( Replace CrossAttention.forward to use xformers caching latents. 0%| | 0/31 [00:00<?, ?it/s]

I am not sure if I am making a mistake somewhere in the installation, but Ive reinstalled a few times now and used a different dataset and I still get this issue.

[Nezara]

I ran into this issue as well.

As soon as the installation finished, did you just import your previously downloaded models to begin training?

If so, try instead to use the built in model download functionality that comes with the Koyha Web UI, it appears to download some additional dependencies as well the the model you plan to use. It still hangs a little, but does start to move forward after that.

Let me know if this works.

[tekierz]

The same thing is happening to me, I had it download thru the tool

[HustLion]

same here with A100 gpu.

[csmit195]

This is happening to a friend I was helping install, works perfectly on my older 1080, while its struggling on his 3000 series. We used the built-in downloader, I wonder if there is something corrupt in the file, I'll help him replace it later and we will see.

[tekierz]

This is happening to a friend I was helping install, works perfectly on my older 1080, while its struggling on his 3000 series. We used the built-in downloader, I wonder if there is something corrupt in the file, I'll help him replace it later and we will see.

The solution I found was to do a clean install and then not use CUDNN 8.6. Then everything ran smoothly on my 3000 series gpu