Real-time Remote Control of Franka Gripper with Gello

[yiyikkk]

Thank you for bringing up Gello—it's a fantastic tool for teleoperating robots!

I am using Gello to teleoperate a Franka Panda FR3, and everything works well except for the gripper control, which is not real-time. Currently, I have to set a threshold—when Gello's gripper closes beyond a certain point, the Franka executes a grasp command. However, this does not fully replicate the behavior of Gello's gripper.

I noticed that your demo uses a Robotiq gripper. Is the control for Robotiq real-time? I would like the Franka gripper to precisely follow the opening and closing percentage of Gello’s gripper while also enabling grasping functionality. For example:

• When Gello’s gripper closes to 70%, Franka’s gripper should also close to 70%, regardless of whether there is an object to grasp.
• If Gello’s gripper closes to 20%, then Franka’s gripper continues closing until it grasps an object. Franka should maintain the grasp as long as Gello’s gripper remains more closed than the object's width.
• Ideally, the tighter Gello’s grip, the greater the force applied by Franka.

Can the Franka gripper support real-time remote control of both width and force? Looking forward to your response. Thanks!

[Siva552]

I am currently facing some difficulties in my research on using Gello to control the Franka FR3. I notice that you mentioned having well work in teleoperating the Franka Panda FR3 with Gello, can you share the specific methods you used to achieve the control of the Franka FR3 with Gello? Thank you very much. About your question, as I know, we can learn from https://frankaemika.github.io/docs/libfranka.html#non-realtime-commands, all commands related to the Franka Hand are classified as non-realtime commands. All the content for controlling Franka and Franka Hand (such as Polymetis, which Gello depends on) is developed based on libfranka. Therefore, we are unable to achieve real-time control of the Franka Hand until the Franka official supports real-time control of the Franka Hand.

[Jeevesh2009]

Hi Guys, I am trying to setup gello with FR3 I am unable to use polymetis to control the arm. I can run FCI , libfranka examples but the polymetis launch_robot.py is simply not woking. I am using RT-Kernel. If you faced similar issues and solved them Can you please help me.

[Siva552]

Hi Guys, I am trying to setup gello with FR3 I am unable to use polymetis to control the arm. I can run FCI , libfranka examples but the polymetis launch_robot.py is simply not woking. I am using RT-Kernel. If you faced similar issues and solved them Can you please help me.

Of course, but I need more information to fix the problems,or contact me directly

[peterrw]

Hi Guys, I am trying to setup gello with FR3 I am unable to use polymetis to control the arm. I can run FCI , libfranka examples but the polymetis launch_robot.py is simply not woking. I am using RT-Kernel. If you faced similar issues and solved them Can you please help me.

Hi, do you really need Polymetis? As an alternative, I'd recommend to check out the ROS 2 integration for the FR3 that got added to this repo two weeks ago: gello_software/ros2 It provides a Dev-Container that makes it very easy to set things up.

Polymetis does not support the current Franka FR3 model out of the box, only the FER (aka Panda). You should be able to get it to work, but you need to adapt a few things in Polymetis:

• change the libfranka submodule to a version that is compatible with your robot's system version
• probably fixing some dependency issues (I had to remove the version of libprotobuf in the enviroment.yaml)
• The limits (joints, torque, ,...) of the FER and FR3 differ. You need to adapt these in the corresponding configuration YAML files.

You will have a much nicer experience with the ROS 2 integration.

[elianamor]

As an alternative, I'd recommend to check out the ROS 2 integration for the FR3 that got added to this repo two weeks ago: gello_software/ros2 It provides a Dev-Container that makes it very easy to set things up.

I'm trying to use the ROS2 integration for the FR3 and am running into a lot of issues. I can't open Docker on my machine because it has a graphics card, but there are issues manually installing the package,s and CMAKE isn't finding packages. Do you have any advice?

[AndreasJosefGassner]

Hi @elianamor! I just opened a PR that should make it easier to set up your local development environment: 🔗 https://github.com/wuphilipp/gello_software/pull/39

If you're still running into issues, feel free to give it a try and let me know how it goes — your feedback would be really helpful!

[elianamor]

Hi @AndreasJosefGassner It worked! Thank you so much, the joint signs provided weren't correct for our setup, but that was the only issue. Again, thank you so much. We have been fighting with this software for a week.

[AndreasJosefGassner]

Hi @elianamor, thank you for your feedback — the PR has now been merged into main.

@yiyikkk, regarding the original question: unfortunately, real-time control of the Franka Hand is not currently supported. We are aware of this limitation and have it on our roadmap, though we are not able to provide a timeline at this stage. If you require real-time control with Gello, we recommend using a gripper such as the Robotiq 2F-85, which is supported in the ROS 2 integration.