stable-diffusion-webui icon indicating copy to clipboard operation
stable-diffusion-webui copied to clipboard

[Bug]: error when training embedding

Open 2blackbar opened this issue 2 years ago • 3 comments
trafficstars

Is there an existing issue for this?

  • [X] I have searched the existing issues and checked the recent builds/commits

What happened?

Applying cross attention optimization (Doggettx). Training at rate of 0.05 until step 25000 Preparing dataset... 100%|██████████████████████████████████████████████████████████████████████████████████| 10/10 [00:01<00:00, 5.29it/s] 0%| | 0/5891 [00:00<?, ?it/s]Traceback (most recent call last): File "F:\sd\modules\textual_inversion\textual_inversion.py", line 506, in train_embedding scaler.step(optimizer) File "F:\sd\venv\lib\site-packages\torch\cuda\amp\grad_scaler.py", line 336, in step assert len(optimizer_state["found_inf_per_device"]) > 0, "No inf checks were recorded for this optimizer." AssertionError: No inf checks were recorded for this optimizer.

Applying cross attention optimization (Doggettx).

Steps to reproduce the problem

  1. just train embedding

It happens randomly but when it happens its hard to get it working, and i have no clue why its happening so no idea how to make it go away im on win10 gtx 1080ti

What should have happened?

it should train normally cause it trained seconds ago

Commit where the problem happens

https://github.com/AUTOMATIC1111/stable-diffusion-webui/commits/master

What platforms do you use to access the UI ?

Windows

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

none

List of extensions

none

Console logs

none

Additional information

No response

2blackbar avatar Feb 09 '23 23:02 2blackbar

Ok i noticed its related to using particular txt file from template prompts, this should not happen at all, so id call it a bug, i had none.txt selected, it errors out, when i select style.txr it works fine

2blackbar avatar Feb 09 '23 23:02 2blackbar

same error

iqubik avatar Feb 26 '23 21:02 iqubik

same

iceandstorm avatar Mar 06 '23 16:03 iceandstorm

If you are using a Prompt Template txt file, try removing any empty lines. Each line should have a prompt without any blank rows between.

Firedan1176 avatar May 17 '23 03:05 Firedan1176

If you are using a Prompt Template txt file, try removing any empty lines. Each line should have a prompt without any blank rows between.

And if there are no empty lines, just a single one and this error still appears ... ? 🤔

Woisek avatar May 23 '23 11:05 Woisek

Same

ghtg avatar Dec 09 '23 11:12 ghtg