train loss=nan

[shonehang]

I want to train the dataset called VEDAI. It is also a dataset with infrared and corresponding rgb images. My software environment is CUDA9.0 Pytorch1.0 Python3.6 When I run command python trainval_cgan_update.py --dataset pascal_voc --net res101_cgan_update --bs 1 --nw 1 --epochs 15 --cuda --name rgb2thermal_flir --use_tfb My GPU memory is only 12G, if bs is set higher, there will be out of memory. The loss quickly becomes nan I checked the annotations of the dataset, and I think it's no problem. Why am I having this problem? I only put infrared images in the dataset. Should I add corresponding rgb images? If I add visible light images, where should I put these rgb images? Thank you very much.