Requesting official support for Jetson Nano (Jetpack 4.3)

[tkircher]

Now that SDK 1.4 supports Jetson Nano officially, it would be nice to have official ROS package support here. This ROS driver does not build under the repository version of ROS in Jetpack 4.3. The system version of OpenCV is 4.1.1, and a repo version of OpenCV 3.2 dev can't be installed alongside ROS. So in order to get this built, significant steps have to be taken to build ROS from source first. I've confirmed that it does work, but definitely not out of the box. I'm not actually sure what the right solution would be, though.

[ooeygui]

Thank you for the request. We're focusing on deployment blockers at the moment, and will take on new behaviors as time permits. I'm adding the help wanted flag in case someone can get to this sooner.

[tkircher]

Update: Jetpack 4.4 is out now.

[shawncal]

+1 I've hit this as well, with Jetpack 4.4 and OpenCV 4.1.1

There are 2 problems here:

• The ROS "cv_bridge" module, that the K4A ROS driver and others depend on, expects OpenCV 3.2 in ROS Melodic
• The OpenCV driver 4.1.1 that comes with JetPack strangely does not have CUDA support enabled

The quick fix: downgrade to OpenCV 3.2 (not recommended)

• WARNING: this does NOT get you CUDA acceleration, so the frames coming off the Kinect will be CPU-bound and excruciatingly slow
• credit: https://answers.ros.org/question/347754/jetson-nano-comes-with-opencv-411-do-i-need-to-downgrade-to-32-for-melodic/ sudo apt -y --allow-downgrades install libopencv-dev=3.2.0+dfsg-4ubuntu0.1 ...and hold (avoid it being upgraded) with sudo apt-mark hold libopencv-dev

The better fix: Rebuild OpenCV and ROS packages to support OpenCV 4.1.1 and CUDA acceleration

Firstly, rebuild OpenCV with CUDA

# (credit: https://www.jetsonhacks.com/2019/11/22/opencv-4-cuda-on-jetson-nano/)
git clone https://github.com/JetsonHacksNano/buildOpenCV 
cd buildOpenCV
./buildOpenCV.sh |& tee openCV_build.log

Install K4A ROS Driver

sudo apt-get install ros-melodic-tf2-geometry-msgs # an extra package needed by the K4A ROS driver
cd ~/catkin_ws/src
git clone https://github.com/microsoft/Azure_Kinect_ROS_Driver.git

Clone additional ROS dependencies - these must be rebuilt from source to pick up your locally built, CUDA-enabled OpenCV

sudo apt install libogg-dev # Needed to rebuild the following packages

# Clone the package repos. Note: get the **NOETIC** version of these packages (yes, even though we're using ROS Melodic) 
cd ~/catkin_ws/src
git clone https://github.com/ros-perception/image_transport_plugins.git --branch noetic-devel --single-branch
git clone https://github.com/ros-perception/image_common.git --branch noetic-devel --single-branch
git clone https://github.com/ros-perception/vision_opencv.git --branch noetic --single-branch

Make a couple updates to vision_opencv (credit: https://stackoverflow.com/a/63584757) nano ~/catkin_ws/src/vision_opencv/cv_bridge/CMakeLists.txt Change the line find_package(Boost REQUIRED python37) to find_package(BOOST REQUIRED python)

nano ~/catkin_ws/src/vision_opencv/cv_bridge/src/Module.hpp In the function do_numpy_import, change the return type from void* to void, and remove the return nullptr;

Build ROS Packages

cd ~/catkin_ws
catkin_make clean
catkin_make -DCATKIN_ENABLE_TESTING=False -DCMAKE_BUILD_TYPE=Release --force-cmake
catkin_make install

You should now be able use OpenCV 4.1.1 with CUDA acceleration. Try launch the driver with

roslaunch azure_kinect_ros_driver driver.launch

...then subscribe to one or more topics (i.e. in RViz)

To view Jetson Nano's system performance, including CPU and GPU utilization, you can use the tegrastats command-line tool (documentation here sudo tegrastats

[caobingquan6]

+1 I've hit this as well, with Jetpack 4.4 and OpenCV 4.1.1

There are 2 problems here:

• The ROS "cv_bridge" module, that the K4A ROS driver and others depend on, expects OpenCV 3.2 in ROS Melodic
• The OpenCV driver 4.1.1 that comes with JetPack strangely does not have CUDA support enabled

The quick fix: downgrade to OpenCV 3.2 (not recommended)

• WARNING: this does NOT get you CUDA acceleration, so the frames coming off the Kinect will be CPU-bound and excruciatingly slow
• credit: https://answers.ros.org/question/347754/jetson-nano-comes-with-opencv-411-do-i-need-to-downgrade-to-32-for-melodic/ sudo apt -y --allow-downgrades install libopencv-dev=3.2.0+dfsg-4ubuntu0.1 ...and hold (avoid it being upgraded) with sudo apt-mark hold libopencv-dev

The better fix: Rebuild OpenCV and ROS packages to support OpenCV 4.1.1 and CUDA acceleration

Firstly, rebuild OpenCV with CUDA

# (credit: https://www.jetsonhacks.com/2019/11/22/opencv-4-cuda-on-jetson-nano/)
git clone https://github.com/JetsonHacksNano/buildOpenCV 
cd buildOpenCV
./buildOpenCV.sh |& tee openCV_build.log

Install K4A ROS Driver

sudo apt-get install ros-melodic-tf2-geometry-msgs # an extra package needed by the K4A ROS driver
cd ~/catkin_ws/src
git clone https://github.com/microsoft/Azure_Kinect_ROS_Driver.git

Clone additional ROS dependencies - these must be rebuilt from source to pick up your locally built, CUDA-enabled OpenCV

sudo apt install libogg-dev # Needed to rebuild the following packages

# Clone the package repos. Note: get the **NOETIC** version of these packages (yes, even though we're using ROS Melodic) 
cd ~/catkin_ws/src
git clone https://github.com/ros-perception/image_transport_plugins.git --branch noetic-devel --single-branch
git clone https://github.com/ros-perception/image_common.git --branch noetic-devel --single-branch
git clone https://github.com/ros-perception/vision_opencv.git --branch noetic --single-branch

Make a couple updates to vision_opencv (credit: https://stackoverflow.com/a/63584757) nano ~/catkin_ws/src/vision_opencv/cv_bridge/CMakeLists.txt Change the line find_package(Boost REQUIRED python37) to find_package(BOOST REQUIRED python)

nano ~/catkin_ws/src/vision_opencv/cv_bridge/src/Module.hpp In the function do_numpy_import, change the return type from void* to void, and remove the return nullptr;

Build ROS Packages

cd ~/catkin_ws
catkin_make clean
catkin_make -DCATKIN_ENABLE_TESTING=False -DCMAKE_BUILD_TYPE=Release --force-cmake
catkin_make install

You should now be able use OpenCV 4.1.1 with CUDA acceleration. Try launch the driver with

roslaunch azure_kinect_ros_driver driver.launch

...then subscribe to one or more topics (i.e. in RViz)

To view Jetson Nano's system performance, including CPU and GPU utilization, you can use the tegrastats command-line tool (documentation here sudo tegrastats

Thank you very much for your very detailed tutorial. I followed this steps, but I found a problem. That is, the compressed image cannot be displayed normally in rviz. I guess it is because the image_transport_plugins function package is compiled separately, and it is not associated with the rviz installed by the system. Is there a solution to this situation?