iaf icon indicating copy to clipboard operation
iaf copied to clipboard

added learning rate bias correction and corrected m_t

Open dchatterjee172 opened this issue 7 years ago • 1 comments

Added learning rate bias correction and,
m_t = m.assign(tf.maximum(beta2_t * m + eps, tf.abs(grad)))
this line is wrong because if (beta2_t * m+eps)<0 and grad==0 then g_t = v_t / m_t will be undefined. correct line is,
m_t = m.assign(tf.maximum(beta2_t * m,tf.abs(grad)+eps))

dchatterjee172 avatar Feb 18 '18 07:02 dchatterjee172

looks like correction for m_t was not needed. as (beta2_t * m+eps) is always >0

dchatterjee172 avatar Feb 18 '18 07:02 dchatterjee172