Suggestion to add how to implement pre-trained policies.

[bara-bba]

Hi! I'm currently working to implement a policy trained using SAC in mujoco into a real robot. I'm trying to load the two q-functions but I obtain weird result in q_loss and the returns. Any suggestion in how to load correctly the policy? Thanks!

[krzentner]

I would recommend comparing the observation distribution between your real-world environment and simulated environment. The most obvious difficulties in sim2real transfer are due to that mismatch. Note that the docs already describe how to use a pre-trained policy. Continuing to train after transferring from sim2real is an active area of research, and I don't have a firm recommendation for how to achieve it. In particular, a Q function describing a policy's behavior in simulation is likely to over-estimate the performance of that policy on the real environment.