This line is too slow

[roomo7time]

https://github.com/openai/weightnorm/blob/dff0cd132e9c6e0a31b76cb243d47a07e0c453cc/tensorflow/nn.py#L188

I've been re-implementing ImprovedGAN for semi-supervised learning for cifar10 using nn.py. This line is defining a variable by initializing it by a tensor from the build-up graph. Initializing a variable by such a tensor makes it so slow. I think this is basically a tensorflow library issue.

[roomo7time]

I am not sure this is a proper resolution for the above, but I tried the below:

(a) change g = tf.get_variable('g', dtype=tf.float32, initializer=scale_init, trainable=True) by g = tf.get_variable('g', shape=scale_init.get_shape().as_list(), dtype=tf.float32, initializer=tf.constant_initializer(1.), trainable=train_g) and b = tf.get_variable('b', dtype=tf.float32, initializer=-m_init*scale_init, trainable=True) by b = tf.get_variable('b', shape=scale_init.get_shape().as_list(), dtype=tf.float32, initializer=tf.constant_initializer(0.), trainable=True)

(b) add op_init_g = g.assign(scale_init); op_init_b = b.assign(-m_init * scale_init)

(c) change return x_init by with tf.control_dependencies([op_init_g, op_init_b]): return x_init

And do likewise for similar terms. And run the graph corresponding to x_init only once at the beginning of training.