Add zero-copy NVMM support with nvbufsurface (JetPack 5)

[poett1]

Just had some time today/yesterday and updated to the new API changes for nvbufsurface on JP5. I checked this with IMX219 (raw) using Argus and See3CAM_CU135 (raw/jpeg) on JP5.1.2 (Orin NX). I hope someone can check this again with Jetpack 4.6. Currently, I do not have any devices running Jetpack below 5.1.2.

There are two main changes:

1. nvv4l2camerasrc: I updated the code to use nvv4l2camerasrc with v4l2 cameras (that do not support Argus) if input_codec=RAW on Jetson Devices. This will output a surface with NVBUF_LAYOUT_PITCH, which is transformed to NVBUF_LAYOUT_BLOCK_LINEAR in gstBufferManager::Enqueue (thus we need to link against nvbufsurftransform). Also, it seems like eglFrame.planeCount=1 for this source. I commented out the check for this in gstBufferManager::Dequeue to disabled warnings. If you have some other ideas for this, feel free to change it, but it seems to work fine. Sometimes nvv4l2camerasrc outputs bad frames (mostly green with a bar of snow). This also happens with gst-launch-1.0 on my Jetson. Reconnecting the USB camera again seems to solve it.

This can be checked with: video-viewer /dev/video1 --input-codec=raw which will create the pipeline: nvv4l2camerasrc device=/dev/video1 do-timestamp=true ! video/x-raw(memory:NVMM), format=(string)UYVY, width=(int)1280, height=(int)720, framerate=30/1 ! appsink name=mysink sync=false

2. nvvidconv: If ENABLE_NVMM=ON we will always use nvvidconv to rotate the video now, even if the source does not output NVMM. nvvidconv will output NVMM to appsink.

This can be checked with: video-viewer /dev/video1 --input-flip=rotate-180 which will create the pipeline: v4l2src device=/dev/video1 do-timestamp=true ! image/jpeg, width=(int)1280, height=(int)720, framerate=60/1 ! jpegdec name=decoder ! video/x-raw ! nvvidconv flip-method=2 ! video/x-raw(memory:NVMM) ! appsink name=mysink sync=false

And with CSI Cameras: video-viewer csi://0 --input-flip=rotate-180 the pipeline will be: nvarguscamerasrc sensor-id=0 ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, framerate=30/1, format=(string)NV12 ! nvvidconv flip-method=0 ! video/x-raw(memory:NVMM) ! appsink name=mysink

I will be happy to help If any additional changes are necessary to get this into main.

[dusty-nv]

Thank you @poett1! I appreciate this a lot! I will need some time to test this on JetPack 4, 5, and 6 first, or #ifdef it for different versions of JetPack. Otherwise users may randomly have camera issues (which is why over time I have just disabled the NVMM because it was more trouble to maintain than it was worth)

[catch-twenty-two]

@poett1 thank you, cherry-picking this cut my CPU usage in half. (I'm on 5.1.2)

[poett1]

@catch-twenty-two thanks for checking out. Yes, reducing CPU load is one of the main benefits. Another benefit is that it drastically increases the possible frame rate you can feed into your application. I'm using four cameras with 720p/120 FPS in my example code. Without these changes, I reached around 40 FPS output, after applying the changes I was able to achieve the full 120 FPS. @dusty-nv I already fixed some bugs I found after this PR, but I also changed the CMake build to use PkgConfig to find the GStreamer libraries, etc. This was needed to build this repo for a meta-tegra-based Yocto distro. You can check out my PkgConfig branch to see the changes.