lerobot
lerobot copied to clipboard
Added depth frame support for dataset recording/loading
What this does
Add depth frame support for Intel RealSense cameras in ALOHA 2 setup
This PR introduces the following changes:
-
Recording:
- Implemented depth frame recording from Intel RealSense cameras
- Depth frames are saved as .png images, preserving absolute values in uint16 format
-
Loading:
- Added depth frame loading functionality to LeRobotDataset class
- Implemented in a similar manner to image frame loading
(Optional) 3. Encoding attempts:
- Explored options for encoding depth frames into video format like image frames
- Currently, .png images are used as no suitable codec was found to preserve absolute depth values without normalization
- .raw format video was successful, but resulted in larger file sizes compared to original .png images
How it was tested
- With ALOHA 2 setup, recorded and replayed successfully a dataset .
How to checkout & try? (for the reviewer)
In ALOHA config (lerobot/configs/robot/aloha.yaml) for every camera, you want to record depth from, set the "use_depth" parameter to "true", like in this example:
cam_right_wrist:
_target_: lerobot.common.robot_devices.cameras.intelrealsense.IntelRealSenseCamera
camera_index: 130322272631
fps: 30
width: 640
height: 480
**use_depth: false** #set "true" if you want to use the depth camera
And then run the recording:
python lerobot/scripts/control_robot.py record \
--robot-path lerobot/configs/robot/aloha.yaml \
--fps 30 \
--root data \
--repo-id a/aloha_depth_test \
--push-to-hub 0 \
--tags tutorial \
--warmup-time-s 5 \
--episode-time-s 30 \
--reset-time-s 10 \
--num-episodes 2