Question for a Feature: How to Expose Backward Pass Quantities?

[RX-00]

Hello,

For context, I am a PhD student studying the interplay and intersection of MPC and RL. I was wondering what is the best way to go about exposing the iLQG value function and Q functions gradients and Hessians seen in iLQGBackwardPass in backward_pass.h? My end goal is to be able to call these values in the Python API agent.

Forgive my very limited understanding of how grpc works, but would the following idea be the general approach to doing this?

1. Extend the protocol buffer interface with messages like GetBackwardPassResultsRequest and GetBackwardPassResultsResponse in agent.proto.
2. Add the corresponding methods to agent_service.h and agent_service.cc.
3. Add getter functions in backward_pass.h and backward_pass.cc and expose them too in planner.h and planner.cc
4. Rebuild mjpc
5. Modify agent.py to have get_backwardpass_results methods that call a request and get a response from grpc.

I'm unsure of how the details would work on certain steps:

• For step 3, it would be more proper to add the getter functions to the policy.h and policy.cc instead of in the planner? My understanding (which might be wrong) is that we call an agent, which has a planner. The agent calls the planner to create optimal control gains, which is stored in the policy. The agent then calls the policy when it wants to apply those control gains.
• For step 4, would it suffice to just rerun python setup.py install as seen in the Python API installation instructions to now have access to those values from the Python agent?

Thank you in advance for any help and guidance!

[thowell]

@RX-00

The overall approach seems correct.

For step 3, I think it makes sense to add a virtual getter function (eg GetQFunctionDerivatives) to Planner for the Q function derivative terms. Each planner will then override this virtual function. iLQGPlanner::GetQFunctionDerivatives can utilize members in the planner member iLQGPlanner:backward_pass to write the Q function derivative terms. The other planners could write zeros.

For step 4, it should be sufficient to run python setup.py install. If issues are encountered, it might be helpful to delete all of the files generated by the command and then try the command again.

Please feel free to follow up with more questions and good luck with this work!