[Question] Robot fingers tunnel through USD geometry when grabbing

[sashansamarajeewa]

Isaac Lab version: 2.2.X Isaac Sim version: 5.0 Docs ref: https://isaac-sim.github.io/IsaacLab/main/source/how-to/cloudxr_teleoperation.html

I am implementing a CloudXR teleoperation project similar to the documentation above.

I have a furniture object (a drawer) which is composed of 3 separate USDs as drawer box, bottom drawer and top drawer. Goal is to assemble these parts with robot hands (the teleoperated hands).

When the robot hand grabs these furniture USDs, the fingers pass through / tunnel through the furniture objects. They don’t collide robustly.

I experimented with collider type = convex decomposition but colliders do not align well with the visual geometry giving an offset like a “bulge” around mesh. With collider type = SDF Mesh approximation, it visually aligns well but collision is still not good as hands tunnel.

I would like to keep the collider shape close to the actual mesh, so SDF Mesh is the preferred option but collision stability is the problem.

CCD is enabled for the drawer parts and the finger tips. The furniture USDs have both a visual material and a physics material applied.

1. Is there any recommended Isaac Lab / PhysX setting for stable grasping when using SDF Mesh collider approximations?
2. Are there any specific params to be fine tuned in USD.

I have attached one of the USD files and a video clip showing the issue.

drawer_box.usd.txt

https://github.com/user-attachments/assets/6ae91042-1d52-42bf-ae4d-85c039f0af53

Code snippet for asset scene definition.

    drawer_box = RigidObjectCfg(
        prim_path="{ENV_REGEX_NS}/DrawerBox",
        init_state=RigidObjectCfg.InitialStateCfg(pos=[-0.02, 0.44, 1.08], rot=[0.0, 0.0, -0.7071, 0.7071]),
        spawn=UsdFileCfg(usd_path="/workspace/isaaclab/source/isaaclab_assets/isaaclab_assets/assembly/drawer/drawer_box/drawer_box.usd",
                         scale=(2.0, 2.0, 2.0),
                         rigid_props=sim_utils.RigidBodyPropertiesCfg(),
                         articulation_props=sim_utils.ArticulationRootPropertiesCfg(articulation_enabled=False),
                         mass_props=sim_utils.MassPropertiesCfg(mass=10),
                         ),
    )

Code snippet for Visual and Physics materials.

    visual_cfg = PreviewSurfaceCfg(
        diffuse_color=(0.6, 0.8, 0.1),
        emissive_color=(0.0, 0.0, 0.0),
    )

    physics_cfg = RigidBodyMaterialCfg(
        static_friction=2.0,
        dynamic_friction=2.0,
        friction_combine_mode="multiply",
    )

Any guidance is much appreciated.

[RandomOakForest]

Thank you for posting this. Could you please review if the suggestions in #4005 are of any help in your case? Thank you.

[sashansamarajeewa]

I tried the suggestions but those don't seem to fully solve the problem.

I noticed a similar issue has been reproduced with a simpler setup https://github.com/isaac-sim/IsaacLab/issues/3054. It would be great if you could look into this and provide a solution.

[RandomOakForest]

Following up on this, as I just mentioned in #4005, could you set this to min_position_iteration_count=64? We have seen this work for slow movements. See this doc for ref. I will move this issue to our Discussions for follow up.