sd3 train,loss is nan

[chongxian]

when I use this command, the loss is nan,how to solve this problem? Thanks for your help the datasets is small,just 290 images, but loss is nan，I try to set the mixed_precision=bf16 and t5xxl_dtype =bf16,but these settings don't work ,the loss is also nan

[mliand]

t5xxl_dtype=bf16

[chongxian]

t5xxl_dtype=bf16

I try this setting，but it doesn't work

[leonary]

Your loss is equal to nan in the initial stage of training. This should be caused by fp16 precision. Set mixed_precision=bf16, and then do not declare t5xxl_dtype.

[chongxian]

Your loss is equal to nan in the initial stage of training. This should be caused by fp16 precision. Set mixed_precision=bf16, and then do not declare t5xxl_dtype.

It doesn't work ,the loss is nan

[chongxian]

I solve the problem now,but this problem may be the bug of train code

[kohya-ss]

Please remove *_sd3_te.npz files in the training directory, when changing the mixed precision or t5xxl_dtype. It recreates cache files.

[order-a-lemonade]

same problem!!!

Is there anything wrong in my script?

[bananasss00]

same problem!!!

Is there anything wrong in my script?

try use xformers instead sdpa

[vikas784]

i also got the same issue, is anyone able to solve it