lerobot icon indicating copy to clipboard operation
lerobot copied to clipboard

Added depth frame support for dataset recording/loading

Open lyshuga opened this issue 1 year ago • 0 comments

What this does

Add depth frame support for Intel RealSense cameras in ALOHA 2 setup

This PR introduces the following changes:

  1. Recording:

    • Implemented depth frame recording from Intel RealSense cameras
    • Depth frames are saved as .png images, preserving absolute values in uint16 format
  2. Loading:

    • Added depth frame loading functionality to LeRobotDataset class
    • Implemented in a similar manner to image frame loading

(Optional) 3. Encoding attempts:

  • Explored options for encoding depth frames into video format like image frames
  • Currently, .png images are used as no suitable codec was found to preserve absolute depth values without normalization
  • .raw format video was successful, but resulted in larger file sizes compared to original .png images

How it was tested

  • With ALOHA 2 setup, recorded and replayed successfully a dataset .

How to checkout & try? (for the reviewer)

In ALOHA config (lerobot/configs/robot/aloha.yaml) for every camera, you want to record depth from, set the "use_depth" parameter to "true", like in this example:

cam_right_wrist:
  _target_: lerobot.common.robot_devices.cameras.intelrealsense.IntelRealSenseCamera
  camera_index: 130322272631
  fps: 30
  width: 640
  height: 480
  **use_depth: false** #set "true" if you want to use the depth camera

And then run the recording:

python lerobot/scripts/control_robot.py record \
   --robot-path lerobot/configs/robot/aloha.yaml \
   --fps 30 \
   --root data \
   --repo-id a/aloha_depth_test \
   --push-to-hub 0 \
   --tags tutorial \
   --warmup-time-s 5 \
   --episode-time-s 30 \
   --reset-time-s 10 \
   --num-episodes 2

lyshuga avatar Sep 27 '24 12:09 lyshuga