SpectralNorm implementation

[allhailjustice]

I am reading the source code of the spectral normalization's implementation. I think it is not consistent with its original paper's algorithm. I am not an expert on tensorflow so please point out if I am wrong.

The paper's algorithm indicates updating w according to df(w/sigma)/dw. However, the tfa's implementation directly assigns w/sigma to w (as w') and then updates w' according to df(w')/dw'. In my experiment, this implementation leads to exploding loss.

Does anyone have an idea on this? Thanks

[bhack]

There is a WIP refactoring can you take a look at https://github.com/tensorflow/addons/pull/2411?

[korneelvdbroek]

I agree with @allhailjustice

Here is the gist with the unit test for the differentiation which fails for the current tfa.layers.SpectralNormalization implementation.

My earlier fix https://github.com/tensorflow/addons/pull/2411 also fails. I will close that PR.

I am working on a new fix, by defining a new class of KernelVariables which computes the auxiliary vectors u and v upon assignment and which adds w/sigma(w) to the forward pass such that w is correctly differentiated in the backward pass. Reach out if you would like to help.

[milmor]

What's the best way to fix TF spectral norm? It's been more than a year, and this layer still doesn't work.

[seanpmorgan]

TensorFlow Addons is transitioning to a minimal maintenance and release mode. New features will not be added to this repository. For more information, please see our public messaging on this decision: TensorFlow Addons Wind Down

Please consider sending feature requests / contributions to other repositories in the TF community with a similar charters to TFA: Keras Keras-CV Keras-NLP