The grad of training loss with respect to z_{sem} is zero

[ZhangMingKun1]

First, thanks for your excellent work!

I've reproduced the model training process outlined in your project. However, during my inspection of the GaussianDiffusionBeatGans.training_losses method (defined in diffusion/base.py), I noticed that the gradient of the loss function with respect to the z_{sem} variable (passed via model_kwargs['cond']) is consistently zero. This raised concern because a zero gradient implies that z_{sem} is not being utilized in the training loss computation, which would essentially prevent the semantic encoder from training.

Could this possibly be a mistake or a misunderstanding on my part? Is this the same in your training process? If not, do you have any valuable ideas of the possible mistake in my reproduction? I would greatly appreciate any insights or clarifications you might be able to provide.

[Kim-Sung-Hun]

I have the same problem. When using the render function with a trained model for inference, output remains same regardless of the input image, changing only due to xT.

[dsj320]

I would like to understand how the author trained the encoder and updated the gradients. Could someone provide me with this information? I would be very grateful for their assistance