PPO2 clip value loss

[yuhsh24]

If Clip the value to reduce variability during Critic training, why using tf.maximum(tf.square(vpred - R), tf.square(vpredclipped - R)) instead of using tf.minimum. Please tell me the reasons.

[NoelFeiyang]

I have the same question. Also, the clip range tf.clip(vpred-OLDVPRED, -CLIPRANGE, CLIPRANGE) is weird. Probably it should be tf.clip(vpred-OLDVPRED, -tf.abs(OLDVPRED)*CLIPRANGE, tf.abs(OLDVPRED)*CLIPRANGE)??

[rallen10]

Does anyone have further insight on either/both of the questions posed?

[brett-daley]

Issue #445 has some discussion on this

[shtse8]

I have the same question too. Searched for an answer but no luck. Please tell me the reason why it uses maximum instead of minimum.

[shtse8]

Ok. I got the answer now. Because baseline calculate pgloss1 and pgloss2 with negative sign. So that it uses maximum afterward.