huggingface
Isaac Sim integration with LeRobot
It would be great to have realistic simulation training using isaacsim and Isaac lab with LeRobot.
@Cadene That would be great. ! Even I would like to contribute if anything I can do into this..
With pleasure. We would need to port a simulator and a simple task with a gym interface similarly to what we did with PushT (pygame) and Aloha (mujoco)
@Cadene And I think there should be some tutorials are required for training a real industrial cobot like franka or UR along with LeRobot which could be helpful to make it more standardized and useful
This is tangentially related, but I built a tool for generating synthetic data in Isaac SIM. I'm happy to help anyone setting it up.
It outputs lerobot datasets: https://github.com/apockill/pi_at_home/
This is tangentially related, but I built a tool for generating synthetic data in Isaac SIM. I'm happy to help anyone setting it up.
It outputs lerobot datasets: https://github.com/apockill/pi_at_home/
it looks like an interesting project! Does that use Isaac sim to record and save to Lerobot dataset format. How you sync with real robot like so-arm100 , using ROS2 bridge?
Yep! It provides isaac extensions and an environment to record, replay, and render trajectories with domain randomization / augmentation applied. Those rendered trajectories are then convertable to lerobot.
For new robots, there's a base class you implement to read/write joints for your specific robot, then you specify that in configuration. Next you'd import an so-arm100 (or whatever robot) URDF into isaac sim, and the extension should be compatible with your robot!
Hi, I’d appreciate some advice regarding using so-arm100 on IsaacSim.
I'm trying to use the action data from a lerobot dataset to control the SO100 robot in Isaac Sim. However, it doesn't behave correctly, and I suspect the issue might be due to a format mismatch.
Here's what I’ve done so far:
- I have followed the instruction to import so-arm100 and made the ros2 node to send the action.
https://wiki.seeedstudio.com/lerobot_so100m_isaacsim/
- I assumed the action data in the dataset is in degrees, so I converted it to radians. Then, I assigned the converted values to JointState.position. But when I run it in Isaac Sim, the resulting motion is completely off.
Does anyone have any suggestions on what might be wrong or what I should check?
Thanks in advance!
Maybe you can use urdf file in here:
https://github.com/TheRobotStudio/SO-ARM100/issues/54
This is tangentially related, but I built a tool for generating synthetic data in Isaac SIM. I'm happy to help anyone setting it up.
It outputs lerobot datasets: https://github.com/apockill/pi_at_home/
Excellent work, but I'm having some problems configuring docker due to network issues, is it convenient for you to share your docker image as a .tar on a net drive or something, thanks a lot!
Hi！Everyone！We just opensource this！ 🚀 LeIsaac — Open-source workflow combining LeRobot x IsaacSim!
Lightwheel has developed LeIsaac, a complete and reproducible workflow that demonstrates how to fine-tune GR00T N1.5 using data collected entirely from simulation — and deploy it seamlessly on real robots!
💡 What’s included? 🏠 SimReady assets — Fully prepared kitchen environment, LeRobot SO101, oranges, and more. 🎮 LeRobot teleoperation data collection in simulation — Integrated end-to-end pipeline for data collection directly in simulation. 🧠 GR00T N1.5 fine-tuning and deployment — Train GR00T N1.5 with simulation data, then deploy and validate on real hardware.
⭐ In the demo video below, the entire process is completed in just 5 simple steps.
👉 GITHUB: https://github.com/LightwheelAI/leisaac
