Input perturbation by increasing the noise strength

[pengzhangzhi]

Hi @forever208 , Great work! I like your observation of the inconsistency between training and sampling and propose a simple input perturbation to mitigate/align it. A question about your implementation code.

new_noise = noise + gamma * th.randn_like(noise)  # gamma=0.1 

I understand this equation as increasing the noise strength because the new noise is essentially two Gaussian noises added with weight 1 and gamma. Thus the resultant new_noise is a gaussian noise scaled by (1+gemma)

If so, can I understand the input perturbation as interpolating with a larger noise?

Thanks, Zhangzhi