iaf
iaf copied to clipboard
added learning rate bias correction and corrected m_t
Added learning rate bias correction and,
m_t = m.assign(tf.maximum(beta2_t * m + eps, tf.abs(grad)))
this line is wrong because if (beta2_t * m+eps)<0
and grad==0
then g_t = v_t / m_t
will be undefined. correct line is,
m_t = m.assign(tf.maximum(beta2_t * m,tf.abs(grad)+eps))
looks like correction for m_t was not needed. as (beta2_t * m+eps) is always >0