Add xarm6 robot with different grippers

[dwaitbhatt]

Add xarm6 with left and right allegro grippers, xarm gripper, and a test version without a gripper. This resolves the request in #635

[StoneT2000]

Thank you for the contribution @dwaitbhatt, I'll try and get around reviewing this later today or next monday!

I think only thing I need to verify really is to check if the new files aren't too large (if they are, we just make them downloadable instead of being on git), and if the simulation speed + collision meshes are reasonable.

[dwaitbhatt]

Thank you for the contribution @dwaitbhatt, I'll try and get around reviewing this later today or next monday!

I think only thing I need to verify really is to check if the new files aren't too large (if they are, we just make them downloadable instead of being on git), and if the simulation speed + collision meshes are reasonable.

Sounds good @StoneT2000! Some notes from my own testing:

• xarm6 with the default xarm gripper (xarm6_xarmgripper) is somehow not stable in simulation - when running with random actions, the gripper joints tend to go out of limits and it eventually leads all robot joints going blowing up and going to nan - similar to #633, although xarm6 takes much longer than 5 steps to become unstable. I suspect the issue there is also with the ability hand, not the arm. In my case, I am using the gripper URDF from the manufacturer's own repo.

• xarm6 with both allegro hands is stable, but for some reason the simulation is faster with CPU (60fps) than GPU (~20fps), atleast when tested with demo_robot_script and random actions.

[StoneT2000]

Thanks for the details.

Another question, for the actual joint controller of the xarm6 gripper, does the end effector have 1 joint (like a mimic joint) or are there multiple controlling

I recall that to correctly model that gripper we need to model the mimic joints correctly and correctly model that in sim.

Finally where are the urdf / assets from? Usually need to add a readme somewhere just for record keeping about where it's from and what modifications are made if any to the original URDF.

[dwaitbhatt]

Another question, for the actual joint controller of the xarm6 gripper, does the end effector have 1 joint (like a mimic joint) or are there multiple controlling

I recall that to correctly model that gripper we need to model the mimic joints correctly and correctly model that in sim.

It does have a single mimic joint to control the gripper, however the implementation is from the manufacturer's repo and not my own. Need to look into this more carefully.

Finally where are the urdf / assets from? Usually need to add a readme somewhere just for record keeping about where it's from and what modifications are made if any to the original URDF.

Here are the sources for the assets and the modifications on top:

1. xarm6_xarmgripper:

   • Source: xArm manufacturer's repo
   • Modifications:
     • Removed gazebo/ros related tags
     • Added safety_controller for safe operation beyond joint limits (inspired by panda description)
     • Added a material (specular/ambient/diffuse color) based on provided MTL files

2. xarm6_nogripper:

   • Source: Same as xarm6_xarmgripper, without gripper links and joints

3. xarm6_allegro_left:

   • Source: urdf from a previous issue, which was confirmed to work
   • Modifications:
     • Updated collision geometries to match xarm6_xarmgripper
     • Updated orientation of gripper so that the gripper wrist is connected to final link of the arm

4. xarm6_allegro_right:

   • Source: Arm description from xarm6_nogripper with right hand allegro gripper description from Maniskill assets

[StoneT2000]

Thanks for the info. RE the gripper modelling, I have maybe the correct code for how to model that which was uploaded in response to #477, its the floating version of robotiq gripper. It is a bit non-trivial though: https://github.com/haosulab/ManiSkill/blob/main/mani_skill/agents/robots/floating_robotiq_2f_85_gripper/floating_robotiq_2f_85_gripper.py

[StoneT2000]

noticed a small issue here. Can the red dot visual be removed? We can keep the invisible eef link.

Another is that is the original URDF meant to be all silver/gray colored? I think the xarm gripper joints are partially black.

[akuramshin]

@StoneT2000 @dwaitbhatt Hello! I loaded the embodiment xarm6_xarmgripper into the StackCube-v1 environment using the demo_manual_control.py script (using pd_ee_delta_pose as the default controller).

Proceeding to press the f/g keys to open/close the gripper causes the robot's ee pose to change despite only changing the gripper position. out2.webm

The reason I am here is because I have been experiencing something similar with my agent setup of the robotiq 2F-85 gripper on the panda arm: out3.webm

In both videos I am just periodically pressing either the f or g key.

Please let me know if this belongs in its own post. I just thought it was relevant since this PR also has the unexpected behaviour.

[StoneT2000]

Hi @akuramshin , how did you implement the robotiq one? Did you set the drives correctly / are you using our URDFs in this repo?

[akuramshin]

I followed your thread here: https://github.com/haosulab/ManiSkill/issues/477#issuecomment-2299807384

Basically replacing the original panda hand with the urdf of the robotiq gripper you provided. For the controller I copy pasted the controller implemented for the FloatingRobotiq2F85Gripper agent (including the drive creation).

[StoneT2000]

Ok for your issue @akuramshin can you create a different issue on this repo? This PR will just be for the xarm gripper which is different. I will follow up there.

[dwaitbhatt]

Hey @StoneT2000! I have made the suggested changes, here are some details:

• xarm_gripper geometry is quite different from robotiq_2f_85 (xarm_gripper has no inner finger or finger pads), and we need to find new "magic arrays" with precomputed poses for drive creation. I wasn't sure exactly which poses are required so I have simply replaced the xarm_gripper with robotiq for now, since they are functionally similar - both are one DoF parallel grippers.
• I added xarm6 SRDF from the manufacturer's repo, however I could not find official SRDFs for allegro and robotiq grippers, so I wrote them myself since there were too many unnecessary collision checks happening there.

Unresolved issues:

• Running demo_robot_script with xarm6_robotiq and --none-actions on GPU backend it throws a critical error and stops responding when I try to set a joint position target. This does not happen with CPU backend. Seems like it is related to _after_loading_articulation() for gripper modeling.
  [2024-10-24 00:05:29.936] [SAPIEN] [critical] PxArticulationJointReducedCoordinate::setDriveTarget(): it is illegal to call this method if PxSceneFlag::eENABLE_DIRECT_GPU_API is enabled!
• The simulation speed issue persists even with the SRDFs, where the simulation is slower with GPU and fast with CPU. This can be prominently seen by running demo_robot_script with xarm6_allegro and --random-actions. Actually this can be seen even without xarm6, when running demo_robot_script with allegro_hand_right on CPU vs GPU backend.
• Let me know if you think I should make assets downloadable.

[StoneT2000]

Sorry I am a little bit behind on the PRs here, I will try to take a deeper look into this later this week.

[dwaitbhatt]

Sorry I am a little bit behind on the PRs here, I will try to take a deeper look into this later this week.

Hello @StoneT2000! Did you get a chance to review this PR?

[StoneT2000]

@dwaitbhatt did you change the gripper? I just checked and its not the old xarm gripper. The robot iq one version seems to work fine although i just ran pick cube with the xarm 6 robotiq gripper and it is not working. Currently debugging

https://github.com/user-attachments/assets/400c15ec-4d6a-4384-99c9-89cc72d6faa3

It is a older version of ManiSkill however, I will try to merge the main branch, which includes some optimizations and also a much better PPO.

Re sim speed there are a lot of extra contacts that can't ever happen, am removing them now with a .srdf file.

[StoneT2000]

Oh I find the reason why pick cube wasn't solvable.

• For stability in grasping the gripper friction had to be bumped up to 1.0. This one I don't really know how to explain unfortunately.
• The is_grasping function is slightly wrong. The fingers being checked are not the pads (their the link behind the pads) and the link pose direction is reversed for some reason.
• Another thing is the is_static condition is a bit different than panda in that you need to ignore all of the gripper links, passive and controlled ones. Otherwise in PickCube robot doesn't learn to sit still.
• The implemented xarm moves slower relative to the panda arm. This is really not a bug or issue, but it means the default table top environments likely have max episode steps set too low (they are optimized for the default robot used to solve the task).
• The slower speed of the xarm (which can be fixed if you bump up the max delta joint positions in the controller) causes RL to be a bit harder although not impossible.

python ppo_fast.py --env_id="PickCube-v1" --seed=1    --num_envs=4096 --num-steps=16 --update_epochs=8 --num_minibatches=32     --total_timesteps=50_000_000     --num_eval_envs=16     --save-model --cudagraph
s --exp-name="ppo-PickCube-v1-state-xarm6-1-walltime_efficient" --num_eval_steps=100

https://github.com/user-attachments/assets/c47c5e43-ee11-4fb2-a171-618ff24ad905

Anyway I think this PR is very close to be ready, there are just a few other minor fixes. I made a branch based on this PR with the fixes I have made: https://github.com/haosulab/ManiSkill/pull/820

The other thing is since we already have a few arms I will likely move xarm6 assets to a seperate huggingface/github repo that users can download. I will likely also not include the allegro hand implementation in the public ManiSkill repo in the new PR primarily because I haven't tested it thoroughly (and would like to split the PR up into smaller pieces for now).

Thanks again for all the initial efforts!

[dwaitbhatt]

@dwaitbhatt did you change the gripper? I just checked and its not the old xarm gripper. The robot iq one version seems to work fine although i just ran pick cube with the xarm 6 robotiq gripper and it is not working. Currently debugging Re sim speed there are a lot of extra contacts that can't ever happen, am removing them now with a .srdf file.

@StoneT2000 Yes, I had changed the gripper because of issues with drive creation for xarm_gripper. I had also written SRDFs for xarm6_robotiq and xarm6_allegro. Details in https://github.com/haosulab/ManiSkill/pull/636#issuecomment-2434495095.

Regarding the rest, thank you for the fixes! Glad to see xarm6 finally learning in Maniskill. Let me know if you need any help with the final xarm6 PR.

[StoneT2000]

closed in favor of #820

[55joe]

This may solve the xarm original gripper problem with magic array got from https://github.com/haosulab/cvpr-tutorial-2022/tree/master?tab=readme-ov-file. Note that this agent also added a tidybot++ franka base. Franka_xarm7.tar.gz

https://github.com/user-attachments/assets/2ba689aa-3d88-4806-9661-77eadc5ec733