robot_init_options: Confusion about head tilt

[Luca-Wiehe]

I was playing around with the initial scene configuration. I have set fixed positions for the objects to be placed. Now, I am adjusting some values in the qpos array that can be passed as robot_init_options but has some provided default values in base_env

The default values and ranges look as follows:

qpos = np.array(
                [-0.2639457174606611, # joint_torso [-4.49, 1.35]
                0.0831913360274175, # joint_shoulder [-2.66, 3.18]
                0.5017611504652179, # joint_bicep [-2.13, 3.71]
                1.156859026208673, # joint_elbow [-2.05, 3.79]
                0.028583671314766423, # joint_forearm [-2.92, 2.92]
                1.592598203487462, # joint_wrist [-1.79, 1.79]
                -1.080652960128774, # joint_gripper (arm joint) [-4.49, 1.35]
                0, 0, # joint_finger_right / joint_finger_left [-0.0001, 1.3]
                -0.00285961, # joint_head_pan [-3.79, 2.22]
                0.7851361 # joint_head_tilt [-1.17, 1.17]
                ]
)

This results in the following scene:

When changing the value for the joint_head_pan, I would expect the robot to rotate its head horizontally, resulting in a different crop of the image.

qpos = np.array(
                [-0.2639457174606611,
                0.0831913360274175,
                0.5017611504652179,
                1.156859026208673,
                0.028583671314766423,
                1.592598203487462,
                -1.080652960128774,
                0, 0,
                -3.5, # only value that changed
                0.7851361
                ]
)

However, the scene doesn't change at all. The robot arm and objects disappear. Why is that the case?

[xuanlinli17]

It's because you are using visual matching. Visual matching always overlays a fixed background png into the robot observation (the table in this case). To see the effect of adjusting robot qpos, try the variant aggregation setup or https://github.com/simpler-env/ManiSkill2_real2sim/blob/ef7a4d4fdf4b69f2c2154db5b15b9ac8dfe10682/mani_skill2_real2sim/examples/demo_manual_control_custom_envs.py (remove rgb_overlay_path)