isaac_ros_dnn_stereo_depth
isaac_ros_dnn_stereo_depth copied to clipboard
Hardware-accelerated, deep learned stereo disparity estimation
Isaac ROS DNN Stereo Depth
Hardware-accelerated, deep learned stereo disparity estimation
Webinar Available
Learn how to use this package by watching our on-demand webinar: Using ML Models in ROS 2 to Robustly Estimate Distance to Obstacles
Overview
Isaac ROS DNN Stereo Depth provides a GPU-accelerated package for DNN-based
stereo disparity. Stereo disparity is calculated from a
time-synchronized image pair sourced from a stereo camera and is used to
produce a depth image or a point cloud for a scene. The isaac_ros_ess
package uses the ESS DNN
model
to perform stereo depth estimation via continuous disparity prediction.
Given a pair of stereo input images, the package generates a disparity
map of the left input image.
ESS is used in a graph of nodes to provide a disparity prediction from an input left and right stereo image pair. Images to ESS need to be rectified and resized to the appropriate input resolution. The aspect ratio of the image is recommended to be maintained, so the image may need to be cropped and resized to maintain the input aspect ratio. The graph for DNN encode, DNN inference, and DNN decode is included in the ESS node. Inference is performed using TensorRT, as the ESS DNN model is designed with optimizations supported by TensorRT. ESS node is agnostic to the model dimension and disparity output has the same dimension as the ESS model.
Isaac ROS NITROS Acceleration
This package is powered by NVIDIA Isaac Transport for ROS (NITROS), which leverages type adaptation and negotiation to optimize message formats and dramatically accelerate communication between participating nodes.
Performance
Sample Graph |
Input Size |
AGX Orin |
Orin NX |
x86_64 w/ RTX 4060 Ti |
---|---|---|---|---|
DNN Stereo Disparity Node Full |
576p |
78.8 fps 4.1 ms |
27.2 fps 6.2 ms |
204 fps 4.4 ms |
DNN Stereo Disparity Node Light |
288p |
288 fps 5.6 ms |
128 fps 5.6 ms |
350 fps 4.2 ms |
DNN Stereo Disparity Graph Full |
576p |
74.0 fps 20 ms |
26.1 fps 42 ms |
191 fps 11 ms |
DNN Stereo Disparity Graph Light |
288p |
260 fps 13 ms |
116 fps 16 ms |
350 fps 12 ms |
Documentation
Please visit the Isaac ROS Documentation to learn how to use this repository.
Packages
Latest
Update 2023-10-18: Updated for ESS 3.0 with confidence thresholding in multiple resolutions.