Robot gripper cannot grasp cloth (slipping problem)

[vrg15101]

slipping.webm I tried to reduce the substeps and tried doing both force and position control for the end effector and it does not seem to help. The gripper is not able to pick up the cloth and maintain constant contact.

[zswang666]

Looks like there is coupling force between the gripper and the cloth as there is one particle moves seemingly due to the gripper. Could you provide the script you are running?

[vrg15101]

Thank you for your response, here is the code,

import genesis as gs
import numpy as np
import IPython
#
########################## init ##########################
gs.init()

########################## create a scene ##########################

scene = gs.Scene(
    sim_options=gs.options.SimOptions(
        dt=0.01,
        substeps=5,
    ),
    coupler_options = gs.options.CouplerOptions(rigid_pbd = True,),
    rigid_options=gs.options.RigidOptions(enable_collision=True,),
    pbd_options=gs.options.PBDOptions(
    particle_size = 0.01,
    gravity = (0,0,-9.8),
    ),
    viewer_options=gs.options.ViewerOptions(
        camera_fov=30,
        res=(1280, 720),
        max_FPS=60,
    ),
    show_viewer=True,
)

########################## entities ##########################
plane = scene.add_entity(
    morph=gs.morphs.Plane(),
)

cloth_1 = scene.add_entity(
    material=gs.materials.PBD.Cloth(),
    morph=gs.morphs.Mesh(
        file="meshes/cloth.obj",
        scale=2.0,
        pos=(0, 0, 0.01),
        euler=(0.0, 0, 0.0),
        collision = True,
    ),
    surface=gs.surfaces.Default(
        color=(0.2, 0.4, 0.8, 1.0),
        vis_mode="particle",
    ),
)
franka = scene.add_entity(
        gs.morphs.MJCF(
        file="xml/franka_emika_panda/panda.xml", 
        pos = (-0.5,0.0,0.0),
        ),
        material=gs.materials.Rigid(coup_friction=4.0),
    )


########################## build ##########################
scene.build()

cloth_1.set_particle_position(0,np.array([0.0,0.0,0.25], dtype=np.float32))
scene.step()
#cloth_1.fix_particle(0)
cloth_1.fix_particle(cloth_1.find_closest_particle((1, 1, 0.0)))
velocity = np.array([0.10, 0.10, 0.10], dtype=np.float32)
scene.step()
fixed_point_pos = cloth_1.get_particles()[0]
fixed_point_pos[2] =fixed_point_pos[2] +0.2
####IPython.embed()

motors_dof = np.arange(7)
fingers_dof = np.arange(7, 9)

franka.set_dofs_kp(
    np.array([4500, 4500, 3500, 3500, 2000, 2000, 2000, 100, 100]),
)
franka.set_dofs_kv(
    np.array([450, 450, 350, 350, 200, 200, 200, 10, 10]),
)
franka.set_dofs_force_range(
    np.array([-87, -87, -87, -87, -12, -12, -12, -100, -100]),
    np.array([ 87,  87,  87,  87,  12,  12,  12,  100,  100]),
)

end_effector = franka.get_link('hand')

qpos = franka.inverse_kinematics(
    link = end_effector,
    pos  = fixed_point_pos,#np.array([-0.75, 0.0, 0.5]),
    quat = np.array([0, 1, 0, 0]),
)
qpos[-2:] = 0.04
path = franka.plan_path(
    qpos_goal     = qpos,
    num_waypoints = 200, # 2s duration
)

for waypoint in path:
    franka.control_dofs_position(waypoint)
    scene.step()

for i in range(100):
    scene.step()

reach_pos = cloth_1.get_particles()[0]
reach_pos[2] = reach_pos[2]+0.085
qpos = franka.inverse_kinematics(
    link = end_effector,
    pos  = reach_pos,
    quat = np.array([0, 1, 0, 0]),
)

franka.control_dofs_position(qpos[:-2], motors_dof)
for i in range(100):
    scene.step()

franka.control_dofs_position(qpos[:-2], motors_dof)
franka.control_dofs_position(np.array([0.001, 0.001]), fingers_dof)  # you can use position control
#franka.control_dofs_force(np.array([-0.5, -0.5]), fingers_dof)

for i in range(200):
    scene.step()

for i in range(300):
    qpos = franka.inverse_kinematics(
        link=end_effector,
        pos=np.array([0.0, 0.0, reach_pos[2] + 0.0005 * i]),
        quat=np.array([0, 1, 0, 0]),
    )	
    franka.control_dofs_position(qpos[:-2], motors_dof)
    scene.step()
for i in range(200):
    scene.step()
for i in range(1000):
    scene.step()

[Eirikalb]

This is the exact same problem I have been grapling with in all simulators I have tried to date: Omniverse, Mujoco, + many others. There is not a single good cloth+robot simulator implementation out there yet which is why alot of AI research on cloth manipulation has been forced to estimate gripping by attaching the cloth rather than actually gripping it.

It would be so awesome if this could be fixed in this work!

[zhouxian]

This is the exact same problem I have been grapling with in all simulators I have tried to date: Omniverse, Mujoco, + many others. There is not a single good cloth+robot simulator implementation out there yet which is why alot of AI research on cloth manipulation has been forced to estimate gripping by attaching the cloth rather than actually gripping it.

It would be so awesome if this could be fixed in this work!

Thank you for the suggestion! We are looking into it and possibly will come up with a better coupling strategy to improve the grasping stability. Rest assured, wherever there’s an issue, we will try our best to fix it. :)

[Louis2099]

Any followup on this issue, I'm also trying to do robot cloth interaction,

which is why alot of AI research on cloth manipulation has been forced to estimate gripping by attaching the cloth rather than actually gripping it.

or is there a way to just attach the deformable object to the robot?

[anashoussaini]

Any news on this issue?

[duburcqa]

Somehow. There is now a way to just "just attach the deformable object to the robot" (see PR#1309). Yet, still not "physical" interaction between cloths and rigid entities for now.

[duburcqa]

Fixed by https://github.com/Genesis-Embodied-AI/Genesis/pull/1767