GP-GAN icon indicating copy to clipboard operation
GP-GAN copied to clipboard

loss is nan

Open 1162197376 opened this issue 1 year ago • 1 comments

Hello author

When I was training, the loss was nan. After debug, I found that all the variables changed to 0 after the first convolutional layer. After the second convolutional layer, some data number became particularly large or small. In the subsequent convolutional layers, some data number would change to 0 or nan.

After remove batchnorm, set the learning rate to 0 and check W of convolutional layers, still not solve this problem. I really need your help. Thank you very much

1162197376 avatar Nov 09 '23 07:11 1162197376

  1. Do you run on your own data?
  2. Can you run inference correctly?
  3. Is the version of chainer match? I think it's likely that the updated chainer is different that the old one.

wuhuikai avatar Nov 09 '23 07:11 wuhuikai