Line search does not check KL constraint satisfaction

[zhihanyang2022]

The TRPO paper (Appendix C) claims that "we use a line search to ensure improvement of the surrogate objective and satisfaction of the KL divergence constraint". However, in the current codebase, the linesearch function only checks whether the candidate update improves the surrogate advantage, which deviates from the paper and doesn't check for KL constraint.

https://github.com/joschu/modular_rl/blob/master/modular_rl/trpo.py#L92

Could you help me understand the reasoning behind this? @joschu

Thanks!