MPRNet icon indicating copy to clipboard operation
MPRNet copied to clipboard

Published 20 hours ago • verified publisher icon swz30

Loss raise to abnormal and batchsize

Open Hong-yu-Zhang opened this issue 2 years ago • 5 comments

Loss raises to several million after 50 epochs (Before 50 epoch is normal). And why I can only allow batchsize 2 on RTX3090 when training, 2 more will out of memory.

Hong-yu-Zhang avatar Nov 09 '22 03:11 Hong-yu-Zhang

I have the same problem. The device I used is the RTX 3090ti.​ After 200 epochs, both the char loss and edge loss grow graduallty.

lianghao2000 avatar Nov 13 '22 03:11 lianghao2000

I'm in the same situation as you. How can I solve it?

jidongkuang avatar Dec 03 '22 11:12 jidongkuang

我和你情况一样。我该如何解决？ clipping the gradient,

torch.nn.utils.clip_grad_norm_(self.net.parameters(), 0.01)

lianghao2000 avatar Dec 03 '22 12:12 lianghao2000

Could you tell me where to put this code？

jidongkuang avatar Dec 03 '22 15:12 jidongkuang

Could you tell me where to put this code？

loss.backward() 
torch.nn.utils.clip_grad_norm_(model_restoration.parameters(), 0.01)
optimizer.step()

lianghao2000 avatar Dec 04 '22 02:12 lianghao2000