Under any parameter, loss is always reduced to nan during training

[177488ZL]

Training under our dataset, img_ size=256 batch_ size=4 or img_ size=128 batch_ Size=16, the final result of all training is that loss is reduced to nan

[Harry-Stephanie]

same problem after 50000 training step

[snippler]

Any idea how to debug this issue?

[Echo-jyt]

Training under our dataset, img_ size=256 batch_ size=4 or img_ size=128 batch_ Size=16, the final result of all training is that loss is reduced to nan

Hi, have you found the solution?

[177488ZL]

Training under our dataset, img_ size=256 batch_ size=4 or img_ size=128 batch_ Size=16, the final result of all training is that loss is reduced to nan

Hi, have you found the solution?

no.....lol

[177488ZL]

no,I tried to train with various parameters for two weeks, and in the end, the loss was always null. My graphics card is 3080ti

At 2023-03-15 20:01:39, "Jin Yuntao" @.***> wrote:

Training under our dataset, img_ size=256 batch_ size=4 or img_ size=128 batch_ Size=16, the final result of all training is that loss is reduced to nan

Hi, have you found the solution?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[Adrian744]

Did you double check your data ?(Make sure that all your data does not contain nan values)

[perkeje]

I had the same issue. If you are training with amp = True, be sure to run the script with accelerate launch script.py. That fixed my problem.