DGMR-pytorch
DGMR-pytorch copied to clipboard
hyungting
TemporalDiscriminator and SpatialDiscriminator
Hello, I recently reproduced this paper, but I encountered a problem. In the discriminator, if I add the BN layer before the full connection layer, my output value will always be a fixed value. Then I use relu (1+x)+relu (1-y), and their value will always be equal to 2. This problem has puzzled me for a long time.Then I tried your discriminator and found that the same situation occurred, which led to my loss in the training process equal to 2 forever
net = TemporalDiscriminator() net.train() x = torch.rand(size=(2,18,256,256)) x = net(x) y = torch.ones(size=(2,18,256,256)) y = net(y) print(nn.functional.relu(1 - x).mean() + nn.functional.relu(1 + y).mean()) tensor(2.0000, grad_fn=<AddBackward0>)
Hi! This is still an unsolved issue for me as well. I was wondering that is it only the temporal discriminators problem or the spatial discriminator also encounters the same problem in your experience.
sorry,I replied to you late. In my experiment, both discriminators have this problem. :(
I checked the gradients of discriminators' weights this morning and found out that almost every weight's gradient is close to zero, except for the bias of the output linear layer. Do you have any idea what may cause this problem? (Note: the gradients of generator looks okay, nothing odd happens in the generator.)
I checked the gradients of discriminators' weights this morning and found out that almost every weight's gradient is close to zero, except for the bias of the output linear layer. Do you have any idea what may cause this problem? (Note: the gradients of generator looks okay, nothing odd happens in the generator.)
This seems to have happened when I did the experiment last month
I checked the gradients of discriminators' weights this morning and found out that almost every weight's gradient is close to zero, except for the bias of the output linear layer. Do you have any idea what may cause this problem? (Note: the gradients of generator looks okay, nothing odd happens in the generator.)
Sorry to finish replying to you. I have too many courses（天天都上网课，老师还要提问）. I think it may be backforward() in Pytorch. You can try to change it to backforward (retain graph=True)
@clearlyzero Hi! I've still training this model recently. I looked into the gradients in the discriminators and concluded that it is mostly the output linear layers' that are dominating the output, so I added nn.Dropout(0.5) before the linear layer to deactivate several neurons while training. This did make an impact on the loss of discriminator. The losses can oscillate around value = 1 after this transformation. However, the loss value still sticks around 1 (e.g. 0.99).
The loss curve of discriminator looks like this right now.
The model is still not well-trained. Since the discriminator is likely to give predictions with low value (e.g. 1e-3), I changed the loss function of discriminator. The value of prediction did become larger (e.g. 0.3~0.8), however, the generator and disrcriminator still seems to be unbalanced at some point. I am now tuning the hyperparameters to make them more balanced. The prediction from the generator of this incomplete version looks like below.
Now I have a situation that the training times have reached a certain number, and the effect is getting worse and worse
Your training result looks great! The fine features seem to be well-captured. Would you like to discuss by email? I assume you can speack mandarin. Maybe we can discuss in private in mandarin. My email is [email protected]
Your training result looks great! The fine features seem to be well-captured. Would you like to discuss by email? I assume you can speack mandarin. Maybe we can discuss in private in mandarin. My email is [email protected] ok
@bugsuse I replaced the hinge loss function with BCE loss function to train the discriminators and finally got a much reasonable result.
@hyungting I have tried using nn.BCELoss to replace the hinge loss, but the result is not reasonable.
I changed the line: https://github.com/hyungting/DGMR-pytorch/blob/bf65bc746180f3c6ced6f3f1bdd95c72dc9e7544/DGMR/DGMRTrainer.py#L44 to
self.discriminator_loss = lambda x, y: nn.BCELoss()(x, y)
and the lines: https://github.com/hyungting/DGMR-pytorch/blob/bf65bc746180f3c6ced6f3f1bdd95c72dc9e7544/DGMR/DGMRTrainer.py#L229-L230 to
D_loss = self.discriminator_loss(fake_D_pred, true_D_pred)
Is this the way you replaced it? Or anything else I have overlooked?
When using BCE loss, you should delete the hinge loss and use BCE directly. I tested it here, but the effect is still not particularly good.
criterion = nn.BCEWithLogitsLoss()
s_real = criterion(S_out_real,torch.ones_like(S_out_real)) s_fake = criterion(S_out_fake,torch.zeros_like(S_out_fake)) loss_tmp = t_real+t_fake