Problem with "Denoising Diffusion Implicit Model" - not learning.

[Bejro]

trafficstars

The model proposed in ddim.py does not converge at all. It stays stable in a state that estimates the noise to be ~0 with an error of 0.7979 forever. My quick fix was to drop the AdamW optimizer and use Adam. I am just getting into diffusion models, thus, can somebody explain to me what exactly the problem is?