conv_seq2seq icon indicating copy to clipboard operation
conv_seq2seq copied to clipboard

weight normalization is probably implemented wrong

Open jingjing-gong opened this issue 7 years ago • 1 comments

https://github.com/tobyyouup/conv_seq2seq/blob/78a6e4e62a4c57a5caa9d584033a85e810fd726e/seq2seq/encoders/conv_encoder_utils.py#L59

you are not using V_norm any where except from initialization of variable tensor g. and according to the paper https://arxiv.org/abs/1602.07868, g should be a scalar instead of a variable tensor. correct me if I am wrong.

jingjing-gong avatar Jan 10 '18 18:01 jingjing-gong

The norm of v is also calculated by this line: https://github.com/tobyyouup/conv_seq2seq/blob/78a6e4e62a4c57a5caa9d584033a85e810fd726e/seq2seq/encoders/conv_encoder_utils.py#L75 And I also think tf.div(g, tf.norm(V, axis=0)) should be tf.div(tf.norm(g), tf.norm(V, axis=0)) so that |W| can be calculated implicitly.

SongRb avatar Mar 28 '18 12:03 SongRb