Challenges with OAK-D Stereo Calibration and RTAB-Map Depth Quality

[TKaluza]

We are experiencing issues when using RTAB-Map with an OAK-D camera, particularly concerning stereo calibration and depth data quality.

• Reference Configuration: RTAB-Map standalone software with OAK-D camera using default calibration. This setup functions correctly and serves as our baseline.(GitHub)
• Custom Configuration: We generated hardware-synchronized stereo images with the OAK-D camera and exported the camera's calibration data. Additionally, we performed our own calibration using Kalibr and the RTAB-Map calibration tool.
• Issues Encountered:
• The intrinsic parameters, especially fx and fy, differ significantly from the reference setup.(Luxonis)
• Depth data quality is poor, with apparent fluctuations in calibration, as if the camera is continuously zooming.
• Consequently, RTAB-Map fails to perform VSLAM effectively with our custom recordings.

1. Is it feasible to use Kalibr for calibrating the OAK-D camera with RTAB-Map? If so, which distortion model is recommended?(Luxonis Forum)
2. Are there specific considerations or best practices when calibrating wide-angle cameras for use with RTAB-Map?
3. Is the field of view (FoV) inherently limited in this setup?
4. Are there alternative software tools better suited for generating depth maps with wide-angle lenses? Is it better to use those information as depth map input for rtabmap?

Additional Information:

• Real-time performance is not a requirement for our application; we prioritize maximum quality. • We welcome any suggestions or insights. Please feel free to ask if any details are unclear.

Related Pull Request:

https://github.com/introlab/rtabmap/pull/1034

[matlabbe]

Hi,

1. Probably. We support standard calibration models, matching OpenCV's distortion parameters. In particular: https://github.com/introlab/rtabmap/blob/6d4e8a4173533fb1340d872ce35dccb5f4ec826d/corelib/src/CameraModel.cpp#L282

   • Note that if cols=4 and model name is equidistant or fisheye, the fisheye model is used. I think with oak-d wide angle cameras, I think Luxonis uses CALIB_RATIONAL_MODEL model by default.

2. It is either using CALIB_RATIONAL_MODEL model or fisheye model. Note however that getting a good calibration is possible with RTAB-Map's calibration tool, but it is hard to get a good one reliably against many cameras, if we don't reproduce exactly the same motion between the calibrations.

3. The rectification will try to keep as much FOV as it can, but yeah, it often means rectified FOV is smaller than raw FOV.

4. You can use any calibration and/or disparity approach you want. On rtabmap side, we just expect rectified images, stereo or RGB-D. If you do the stereo rectification and disparity->depth computation on your side, you can just send the rectified image of the reference camera and the corresponding depth image to rtabmap (with rectified camera_info output).

cheers, Mathieu