Gradient Penalty

[HCA97]

Hi,

I have a question regarding gradient penalty. I believe the L2-norm calculation of the gradient is incorrect. It should be sum across all axes except the batch axis because the norm of a matrix should be a scalar value. However, your L2-norm calculation returns a matrix not a scalar value. Because you are summing across the channel axis only. Is there a special reason for that?

https://github.com/baumgach/vagan-code/blob/6be0b51eff8e5bf2268e7cb766e02f273ed87984/vagan/model_vagan.py#L134

Best Regards, Cem

[rmukh]

@HCA97, I guess you are right. It should be smth like:

slopes = tf.sqrt(tf.reduce_sum(tf.square(grads), axis=[1, 2, 3]))
penalty = self.exp_config.scale * tf.reduce_mean(tf.square(slopes - 1.0))