How to Handle Limited Observability in TDMPC?

[lrchit]

Hi,

Thank you for the incredible work on TDMPC!

I'm implementing it on my own task with the Unitree Go1 robot, and I have some questions regarding the "observation".

In the image above:

• The left represents the "stand task" without privileged information (i.e., the velocity of the base link and the robot's height).
• The right shows the task with these additional privileged observations.

From my experiments, it’s significantly harder for the agent to learn without the privileged information, as shown in the image. After 14 hours of training, the agent on the right still struggles. By contrast, the agent with privileged information (left) stands reliably after just 40 minutes.

This leads to my confusion: Is it inherently too challenging to train TDMPC when only the latent state is used for the value (critic network), which is inferred from only proprioceptive data? While a motion capture system might be available for rewards during training, in my case, the trained policy would only have access to proprioceptive data during deployment on the real robot.

I’m considering a teacher-student framework:

1. The teacher loop is trained first with full access to privileged information to refine the latent states.
2. The student loop then learns to "imitate" the latent states using only proprioceptive data.

Do you think such an approach would help?

Looking forward to your insights!

Best regards, Ruochen Li

[nicklashansen]

Hi Ruochen, sorry that I missed this! I'm not a roboticist so I can't say for sure what's the best pipeline for real robot deployment, but a student-teacher training scheme might solve that problem indeed. For a model-free algo like SAC or PPO there's more separation between policy and value function which makes asymmetric training more straightforward, it's not clear how to best achieve this separation in a model-based context. I would be interest to hear more about your solution if you obtain any results!