pytorch-CycleGAN-and-pix2pix icon indicating copy to clipboard operation
pytorch-CycleGAN-and-pix2pix copied to clipboard

Hello, I am very interested in the work done by this model, but the following situation occurred during GPU training, my graphics card model is RTX3060.

Open wangyeheng123 opened this issue 3 years ago • 2 comments

(epoch: 1, iters: 100, time: 0.317, data: 2.125) D_A: nan G_A: nan cycle_A: nan idt_A: nan D_B: nan G_B: nan cycle_B: nan idt_B: nan (epoch: 1, iters: 200, time: 0.320, data: 0.000) D_A: nan G_A: nan cycle_A: nan idt_A: nan D_B: nan G_B: nan cycle_B: nan idt_B: nan (epoch: 1, iters: 300, time: 0.321, data: 0.000) D_A: nan G_A: nan cycle_A: nan idt_A: nan D_B: nan G_B: nan cycle_B: nan idt_B: nan (epoch: 1, iters: 400, time: 0.428, data: 0.000) D_A: nan G_A: nan cycle_A: nan idt_A: nan D_B: nan G_B: nan cycle_B: nan idt_B: nan (epoch: 1, iters: 500, time: 0.312, data: 0.000) D_A: nan G_A: nan cycle_A: nan idt_A: nan D_B: nan G_B: nan cycle_B: nan idt_B: nan (epoch: 1, iters: 600, time: 0.318, data: 0.000) D_A: nan G_A: nan cycle_A: nan idt_A: nan D_B: nan G_B: nan cycle_B: nan idt_B: nan (epoch: 1, iters: 700, time: 0.323, data: 0.000) D_A: nan G_A: nan cycle_A: nan idt_A: nan D_B: nan G_B: nan cycle_B: nan idt_B: nan (epoch: 1, iters: 800, time: 0.385, data: 0.000) D_A: nan G_A: nan cycle_A: nan idt_A: nan D_B: nan G_B: nan cycle_B: nan idt_B: nan

wangyeheng123 avatar Jul 04 '22 16:07 wangyeheng123

Which training command line did you use? Did you train a model on the default datasets?

junyanz avatar Jul 04 '22 16:07 junyanz

Thanks for your reply, I use the command line is “python train.py --dataroot ./datasets/maps --name maps_cyclegan --gpu_ids 0 --model cycle_gan”. The same error occurs when I use the command "python train.py --dataroot ./datasets/facades --name facades_pix2pix --model pix2pix --direction BtoA". Download is the default data set, built in Windows 10 above the environment. Pytorch version is '1.4.0+ CU92 '. With CPU training, everything is fine, but with GPU acceleration, the number of parameters explodes and this happens. Looking forward to your reply.

wangyeheng123 avatar Jul 05 '22 04:07 wangyeheng123