learning_from_play icon indicating copy to clipboard operation
learning_from_play copied to clipboard

Published 20 hours ago • verified publisher icon sholtodouglas

Fix VAE training stability

Open 0xtristan opened this issue 4 years ago • 0 comments

Currently @sholtodouglas uses dynamic gradient clipping (gradient clipped if 4x the previous norm). This still seems to cause some issues with loss spikes.

Other options are static clipping, gradient penalties, weight regularisation/normalisation/decay

A personal favourite of mine that worked very well (and is also leveraged in NVAE) is SpectralNorm. TFA has an implementation here: https://www.tensorflow.org/addons/api_docs/python/tfa/layers/SpectralNormalization

0xtristan avatar Jan 20 '21 21:01 0xtristan