[Proposal] Checklist of environments to add to the framework

[Mayankm96]

Proposal

This issue is a tracking page for our current plan to add environments to the Isaac Lab framework.

Motivation

The initial Orbit work mentioned a suite of rigid and deformable body simulation environments. However, many of these environments have been held back due to breaking changes in the framework APIs, poor simulation stability and scaling (particularly for deformable bodies and particle-based cloths), and a lack of resources to transfer all the implementations. Since this is a repeated topic of discussion on GitHub, I am creating this issue to keep track of the environments and provide a (very rough) outline/roadmap.

We are actively working on adding new features and improvements. It is hard to dedicate time to everything at once, but slowly, we are adding new environments as well. If you're interested in contributing, feel free to contact us!

Checklist

Completed ✅

For a complete list, please check here: https://isaac-sim.github.io/IsaacLab/source/features/environments.html

• [x] Reach - Tracks a pose command for an end effector
• [x] Lift - Lifts the object to the desired position
• [x] In-Hand Repose - Re-orient a cube with a dexterous hand
• [x] Cabinet - Open or close a cabinet (hinged object)
• [x] Quadrupedal Locomotion - Track velocity command over a variety of terrain with a legged robot
• [x] Quadrupedal Navigation - Track desired position command with a legged robot

In Progress 🏗️

• [x] Nut-bolt - Tighten the Nut on the specified Bolt
• [ ] Soft Lifting - Lift a soft object to the target position: https://github.com/isaac-sim/IsaacLab/pull/842
• [x] Throwing and catching - Throw and catch a ball with two floating dexterous hands
• [ ] Soft Stacking - Stacking soft objects together

Help Needed 🆘

• [ ] Beat the Buzz - Move a key around a pole without touching it
• [ ] Pyramid Stack - Build blocks into a pyramid
• [ ] Hockey - Shoot the puck into the net with the stick
• [ ] Peg In Hole - Insert the block into the hole
• [ ] Jenga - Take out blocks and stack them into towers
• [ ] Soft Placing - Placing soft objects on a shelf
• [ ] Soft Dropping - Dropping soft objects into a container
• [ ] Tower of Hanoi - Stacks of soft objects around pillars
• [ ] Rope Reshaping - Reshape the rope on the table

Stand-by 🛑

• [ ] Cloth Lifting - Lift the cloth to the desired position
• [ ] Cloth Folding - Fold the cloth into the desired state
• [ ] Cloth Spreading - Spread cloth across the table
• [ ] Cloth Dropping - Place the cloth in the container
• [ ] Flag Hoisting - Raise a flag on a table
• [ ] Fluid Pouring - Pour the liquid into another container
• [ ] Fluid Transport - Move a container full of liquid without causing any spills

[hanschanhs]

Thanks you for the great work - Wanted to ask about the cloth/tshirt prims in IsaacLab. What is the hold up/why is it listed as standby? Separately, If I just want to spawn a tshirt, what is the best way to do this currently in Isaac Lab? Novice question. Thanks for this!

[Mayankm96]

Hi @hanschanhs

We initially experimented with particle-based cloth simulation. However, we had difficulty tuning these cloth properties for smooth gripper and cloth interactions. On further discussions with the PhysX team, we have been informed that an FEM implementation for cloths is under works. Thus, the environments for cloths are in standby as we wait for the nicer solver implementation to be available.

If you wish to spawn a shirt, the easiest is make a USD file with the right properties and then spawn it into the scene through AssetBase class or drive a ParticleBasedCloth class from it yourself with the desired APIs.

[robegi]

Good morning, and thank you for the update! If I may ask, I would like to know something more about the issues with the particle based liquid tasks. I am currently working on a liquid pouring task, and until now I managed to spawn and set up the fluid correctly, but I am a bit worried about the "poor simulation stability and scaling" you mentioned above. Is it possible to train an agent using particle based fluids? Thanks in advance!

[allenzren]

Hi @hanschanhs

We initially experimented with particle-based cloth simulation. However, we had difficulty tuning these cloth properties for smooth gripper and cloth interactions. On further discussions with the PhysX team, we have been informed that an FEM implementation for cloths is under works. Thus, the environments for cloths are in standby as we wait for the nicer solver implementation to be available.

If you wish to spawn a shirt, the easiest is make a USD file with the right properties and then spawn it into the scene through AssetBase class or drive a ParticleBasedCloth class from it yourself with the desired APIs.

Hi @Mayankm96, do you think then it is possible to simulate manipulating a cloth with a Franka arm right now? or the gripper and cloth interaction issue is not possible to solve with the current solver? Thanks!

I am happy to spend time setting this up and tuning the parameters, but not sure if cloth manipulation is fundamentally possible right now.

[wzx16]

Hi @hanschanhs We initially experimented with particle-based cloth simulation. However, we had difficulty tuning these cloth properties for smooth gripper and cloth interactions. On further discussions with the PhysX team, we have been informed that an FEM implementation for cloths is under works. Thus, the environments for cloths are in standby as we wait for the nicer solver implementation to be available. If you wish to spawn a shirt, the easiest is make a USD file with the right properties and then spawn it into the scene through AssetBase class or drive a ParticleBasedCloth class from it yourself with the desired APIs.

Hi @Mayankm96, do you think then it is possible to simulate manipulating a cloth with a Franka arm right now? or the gripper and cloth interaction issue is not possible to solve with the current solver? Thanks!

I am happy to spend time setting this up and tuning the parameters, but not sure if cloth manipulation is fundamentally possible right now.

I have the same question. In addition, I'd like to know what would be the recommended way to spawn a rope prim. Thank you!

[gursi26]

is the repo open to PRs addressing this issue? i assume environments in the Help Needed and Standby section are available to be worked on. I am interested in contributing.

Also, is there a preference between Manager based/Direct workflows for these tasks?

[kellyguo11]

Yes, we'd love to take in contributions for the "Helped Needed" environments. Would be amazing to have both manager-based and direct workflow implementations, but also feel free to choose the workflow you prefer.

[AnnabellaMacaluso]

Hi, I was wondering are there any updates to the rope prim / "Rope Reshaping - Reshape the rope on the table" environment?

[zoemcc]

Hi, I'd like to work on some of the "Help Needed" envs. Is there a more complete description of what the envs used to be lying around anywhere? The ORBIT project's website "code" link redirects to the new Isaac Lab so I'm having trouble finding the descriptions or videos of them, or assets that were used previously. I can recreate my own interpretation of the envs but a little more description would be useful. Are there any of them that are higher priority to the team than others?

I think the most clear one to me is Peg in Hole, so I might start there unless advised otherwise.

Thanks!

[yash-s20]

are any of these environments planned to include rgb / depth cameras? I believe the only example environments that include cameras are Cartpole related