Connecting more than one RealSense not working with RealSenseSensor.init_sensor()

[michezio]

Checklist

• [X] I have searched for similar issues.
• [X] For Python issues, I have tested with the latest development wheel.
• [X] I have checked the release documentation and the latest documentation (for master branch).

Describe the issue

I'm trying to connect 2 RealSense D455 cameras with USB connection. They show up correctly in the RealSense Viewer, and I can connect each of them singularly using init_config(sensor_index=0) on a open3d.t.io.RealSenseSensor object. When I try to connect both at the same time, although they show up correctly using open3d.t.io.RealSenseSensor.list_devices(), using init_config(sensor_index=0) and init_config(sensor_index=1) initializes the same devices twice.

Steps to reproduce the bug

import open3d

open3d.t.io.RealSenseSensor.list_devices()

rs1 = open3d.t.io.RealSenseSensor()
rs2 = open3d.t.io.RealSenseSensor()

rs1.init_sensor(sensor_index=0)
rs2.init_sensor(sensor_index=1)

rs1.start_capture()
rs2.start_capture()

rs1.stop_capture()
rs2.stop_capture()

Error message

[Open3D INFO] [0] Intel RealSense D455: 117222252197
[Open3D INFO]   color_format: [RS2_FORMAT_BGR8 | RS2_FORMAT_BGRA8 | RS2_FORMAT_RAW16 | RS2_FORMAT_RGB8 | RS2_FORMAT_RGBA8 | RS2_FORMAT_Y16 | RS2_FORMAT_YUYV]
[Open3D INFO]   color_resolution: [1280,720 | 1280,800 | 424,240 | 480,270 | 640,360 | 640,480 | 848,480]
[Open3D INFO]   depth_format: [RS2_FORMAT_Z16]
[Open3D INFO]   color_fps: [10 | 15 | 30 | 5 | 60 | 90]
[Open3D INFO]   depth_resolution: [1280,720 | 256,144 | 424,240 | 480,270 | 640,360 | 640,480 | 848,100 | 848,480]
[Open3D INFO]   depth_fps: [100 | 15 | 30 | 5 | 60 | 90]
[Open3D INFO]   visual_preset: []
[Open3D INFO] [1] Intel RealSense D455: 117222251231
[Open3D INFO]   color_format: [RS2_FORMAT_BGR8 | RS2_FORMAT_BGRA8 | RS2_FORMAT_RAW16 | RS2_FORMAT_RGB8 | RS2_FORMAT_RGBA8 | RS2_FORMAT_Y16 | RS2_FORMAT_YUYV]
[Open3D INFO]   color_resolution: [1280,720 | 1280,800 | 424,240 | 480,270 | 640,360 | 640,480 | 848,480]
[Open3D INFO]   color_fps: [10 | 15 | 30 | 5 | 60 | 90]
[Open3D INFO]   depth_format: [RS2_FORMAT_Z16]
[Open3D INFO]   depth_resolution: [1280,720 | 256,144 | 424,240 | 480,270 | 640,360 | 640,480 | 848,100 | 848,480]
[Open3D INFO]   depth_fps: [100 | 15 | 30 | 5 | 60 | 90]
[Open3D INFO]   visual_preset: []
[Open3D INFO] Open3D only supports synchronized color and depth capture (color_fps = depth_fps).
[Open3D INFO] Capture started with RealSense camera 117222252197
[Open3D INFO] Capture started with RealSense camera 117222252197
[Open3D INFO] Capture stopped.
[Open3D INFO] Capture stopped.

Expected behavior

It should initialize and start capture on both cameras and not only the first one twice:

[Open3D INFO] Capture started with RealSense camera 117222252197
[Open3D INFO] Capture started with RealSense camera 117222251231

Open3D, Python and System information

- Operating system: Windows 10 64-bit
- Python version: 3.8.10 (tags/v3.8.10:3d8993a, May  3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)]
- Open3D version: 0.14.1
- System type: x86_64
- Is this remote workstation?: no
- How did you install Open3D?: pip

Additional information

No response

[theNded]

@ssheorey do you happen to have more than 1 RealSense? I don't have the hardware to verify the issue...

[michezio]

For your information, I managed to connect both using two different configuration JSON files with the serial number in it, like this:

{
    "serial": "117222252197",
    "color_format": "RS2_FORMAT_RGB8",
    "color_resolution": "1280,720",
    "depth_format": "RS2_FORMAT_Z16",
    "depth_resolution": "1280,720",
    "fps": "30"
 }

and then using:

with open("./config/realsense_1.json") as cf:
    rs1 = o3d.t.io.RealSenseSensor()
    rs1.init_sensor(sensor_config=o3d.t.io.RealSenseSensorConfig(json.load(cf)))

with open("./config/realsense_2.json") as cf:
    rs2 = o3d.t.io.RealSenseSensor()
    rs2.init_sensor(sensor_config=o3d.t.io.RealSenseSensorConfig(json.load(cf)))

It works, but it's not really flexible

[ssheorey]

@michezio thanks for reporting and finding the workaround. @theNded I have 2 RS cameras, I can take a look at this.

[michezio]

Hi, I'd like to contribute providing the solution I'm currently using to connect 2 or more RealSense camera without manually inserting their serial number.

I've not tested version 0.15 yet so I don't know if this issue is still present, but I see that the files possibly related to this issue have not been updated since version 0.14 so I think the bug is still there.

Anyway, the solution is a function to automatically generate a configuration including the serial number of the device given its index.

def generate_config_for_device(index, config_file=None):
    
    # enumerate and get the serial number of the device at the chosen index
    devices = o3d.t.io.RealSenseSensor.enumerate_devices()
    serial = devices[index].serial
    
    if config_file is not None:
        # if config_file is a string, interpret as the path of a json file
        if isinstance(config_file, str):
            with open(config_file) as cf:
                conf = json.load(cf)
        # else interpret as dictionary in json format
        else:
            assert isinstance(config_file, dict), "config_file is not a dictionary"
            conf = config_file.copy()
        # add serial to the provided configuration
        if "serial" not in conf.keys() or conf["serial"] == "":
            conf["serial"] = serial      
    else:
        # add serial to a default configuration
        conf = {
            "serial": serial,
            "color_format": "RS2_FORMAT_ANY",
            "color_resolution": "0,0",
            "depth_format": "RS2_FORMAT_ANY",
            "depth_resolution": "0,0",
            "fps": "0",
            "visual_preset": "VISUAL_PRESET_DEFAULT"
        }
    
    return o3d.t.io.RealSenseSensorConfig(conf)

By default a "general" configuration is used but one can optionally pass a dictionary in config_file (or the path to a JSON file) so that the serial number is included in that configuration if not already present.

This function can be used in the following way:

import open3d

rs1 = open3d.t.io.RealSenseSensor()
rs2 = open3d.t.io.RealSenseSensor()

configuration = {
    "color_format": "RS2_FORMAT_RGB8",
    "color_resolution": "1280,720",
    "depth_format": "RS2_FORMAT_Z16",
    "depth_resolution": "1280,720",
    "fps": "30"
 }

rs1.init_sensor(sensor_config = generate_config_for_device(0, configuration))
rs2.init_sensor(sensor_config = generate_config_for_device(1, configuration))

This way one can use 2 or more cameras with no need to specify manually their serial number, just like it was intended with the bugged sensor_index argument of init_sensor.

Hopefully it can be useful for anybody in the same situation until an official fix is released.

[ssheorey]

Thanks @michezio for the solution!