conv_seq2seq
conv_seq2seq copied to clipboard
weight normalization is probably implemented wrong
https://github.com/tobyyouup/conv_seq2seq/blob/78a6e4e62a4c57a5caa9d584033a85e810fd726e/seq2seq/encoders/conv_encoder_utils.py#L59
you are not using V_norm
any where except from initialization of variable tensor g
. and according to the paper https://arxiv.org/abs/1602.07868, g should be a scalar instead of a variable tensor. correct me if I am wrong.
The norm of v
is also calculated by this line:
https://github.com/tobyyouup/conv_seq2seq/blob/78a6e4e62a4c57a5caa9d584033a85e810fd726e/seq2seq/encoders/conv_encoder_utils.py#L75
And I also think tf.div(g, tf.norm(V, axis=0))
should be tf.div(tf.norm(g), tf.norm(V, axis=0))
so that |W| can be calculated implicitly.