pytorch-CycleGAN-and-pix2pix icon indicating copy to clipboard operation
pytorch-CycleGAN-and-pix2pix copied to clipboard

Published 20 hours ago • verified publisher icon junyanz

Training Freezes

Open aakash-chakraborty1995 opened this issue 3 years ago • 6 comments
trafficstars

Every time I run pix2pix model with facades dataset my training freezes after 300 iterations of 195 epoch. The total number of epoch is 200 and the gpu memory is available too.

How to complete the training process?

aakash-chakraborty1995 avatar May 06 '22 09:05 aakash-chakraborty1995

Haven't seen this before. Does --continue_train work for you? You can resume the model training from epoch 195.

junyanz avatar Jun 14 '22 19:06 junyanz

I had the same problem. After that I tried --continue_train but it didn't work . Is the problem because of my GPU only 8G? I used 20000 images to train pix2pix , imgsize 512*256(paired img).

raylu1314coding avatar Jul 14 '22 03:07 raylu1314coding

I have the same problem. Do you solve the problem？

xuzhi0413 avatar Jul 22 '22 07:07 xuzhi0413

I have the same problem. Do you solve the problem？

xuzhi0413 avatar Jul 22 '22 07:07 xuzhi0413

It's hard to know. Every case is different. Would it be possible to share with us some console output / debugging output?

junyanz avatar Oct 04 '22 21:10 junyanz

I also have the same problem!

junzhin avatar Dec 19 '23 17:12 junzhin