IsaacLab icon indicating copy to clipboard operation
IsaacLab copied to clipboard

Add support for heterogeneous observations

Open jsmith-bdai opened this issue 1 year ago • 2 comments

  • Add the ability to mix non-homogeneous observations, for example images and with proprioceptive states.

  • Add example task: Cartpole balancing towards a red sphere. Observations: cartpole joint state, RGB image of the scene

jsmith-bdai avatar Aug 23 '24 13:08 jsmith-bdai

Proposal (after some discussions with @kellyguo11): Define the spaces using both: gym.spaces and python basic data types (for simplicity) indistinctly in env configs files as follows. Use the names observation_space, action_space and state_space rather than num_observations, num_actions and num_spaces in configs files to define them respectively.

Fundamental spaces:

gym.spaces python basic data type
Box int or list of int (e.g.: 10, [64, 64, 3])
Discrete set (e.g.: {2})
MultiDiscrete list of sets (e.g.: [{2}, {5}])

Composite spaces:

gym.spaces python basic data type
Dict dict (e.g.: {"joints": 10, "camera": [64, 64, 3], "gripper": {2}})
Tuple tuple (e.g.: (10, [64, 64, 3], {2}))

Toni-SM avatar Sep 15 '24 22:09 Toni-SM

I am not a huge fan of having to manually specify this space since users (in robotics at least) are usually changing the observations and actions on the fly while prototyping the environment. Changing the observation function and then the gym.Space for it seems cumbersome and requires "counting" dimensions (which is very erroneous).

Particuarly, for the manager based environment workflow, we expect users to only provide the "terms" and we infer everything from those terms. Having to specify the numbers for explicit spaces would add more steps for the user (unless we can think of a nicer solution).

But in general, I agree. We should handle spaces somewhat correctly.

Mayankm96 avatar Sep 16 '24 04:09 Mayankm96