Multiple Image Pipeline nodes not compatible with multiple Argus Cameras

[Calvin-InDro]

Jetpack 6.0/6.1 x2 IMX219 x2 isaac_ros_argus_camera mono nodes x2 isaac_ros_image_proc rectification nodes

While able to access any CSI connected IMX219s and stream multiple simultaneously via gst on Jetpack 6.0/6.1, opening multiple isaac_ros_argus_camera mono nodes simultaneously with different cameras results in strange behaviour from the isaac_ros_image_pipeline. When attempting to rectify the images for example, one stream (random if stream #1 or stream #2) will be rectified perfectly, while the other results in an empty image. Sometimes, however, the second stream appears to utilize nonsensical intrinsic camera parameters (while other times, rectifying completely normally). These are low-distortion lenses, and the camera calibrations are accurate, and DO result in perfectly rectified images when applied correctly (plum_bob models).

Pictured here are the rare times the second stream results in a non-empty image with nonsensical rectifications:

Any insight on this behavior would be greatly appreciate.

[rowanborder]

I am also experiencing this issue trying to use 4x IMX296 cameras with argus_camera and the rectify_node. 1 in 10 times all of the rectification nodes will start without issue, but the rest of the time at least one camera produces a black image or a corrupted image, as shown above.

[Infinite-Echo]

Also experiencing a similar issue, but with only one IMX219 camera. It is seemingly random whether it comes up correctly or not.

[sebastian-indro]

Same here, would love Jetpack 6.2 - isaac 3.2 support :)

[sgillen]

Hello, could one of you share the argus logs when this issue occurs?

sudo journalctl -u nvargus-daemon.service

[rowanborder]

I captured the argus logs from two attempts starting the camera driver node, nvidia::isaac_ros::argus::ArgusMonoNode, and the rectify node, nvidia::isaac_ros::image_proc::RectifyNode, - one where a black image was generated and the other where the image was correctly rectified. Unfortunately I can't notice a difference between the two.

I have also previously tried adding a delay between starting the camera driver node and the rectify node, in case there was some kind of race condition that started the rectification before the camera node was properly initialised, but this was unsuccessful.

I have also included the ISAAC ROS logs from the two attempts, in case those are useful, and a video illustrating the issue.

nvargus-daemon_rectify_correct_image.txt nvargus-daemon_rectify_black_image.txt

isaac_ros_logs_rectify_correct_image.txt isaac_ros_logs_rectify_black_image.txt

https://github.com/user-attachments/assets/6d4bf05a-c6e5-4b5a-9d8d-5ca852c421f9

[sebastian-indro]

@sgillen Hi again Sean, is this the info you needed? If something is missing we can provide more Cheers thank you for helping on this issue!

[sgillen]

Hi, yes - @arai2601 could you please take a look at these argus logs? Let's narrow down what layer of the stack the issue is occurring at.

[sgillen]

Arai got back to me offline, we don't see any argus level issues, and I don't see anything obvious from the ros level logs that were shared.

So lets try to narrow this down further, is it just the rectification that has issues or is it the output from the camera node? Are any of you able to replicate this issue without the rectification node? It may also help to get more verbose logs, which we can get with:

export GXF_LOG_LEVEL=info

[rowanborder]

I can confirm that the output from the camera node is correct in both scenarios. I'll try running the tests again with more verbose logging.

[rowanborder]

I ran the tests again with export GXF_LOG_LEVEL=info and have attached the logs. I don't notice any difference in the additional messages though.

isaac_ros_logs_rectify_correct_image_gxf_info.txt isaac_ros_logs_rectify_black_image_gxf_info.txt

[vixp24]

Has anyone successfully resolved this issue? @sgillen I can confirm that the output of the Argus node is always "working" in this failure condition, and so is the output of a downstream resize node. Keep in mind this is viewed in rqt, so it would be subscribing to the sensor_msgs/Image, not the NitrosImage that RectifyNode would be using. I've also found that when ArgusMonoNode is replaced with a gscam2 node, the RectifyNode does not exhibit this behaviour.. Any thoughts?

I've documented my findings here.

Thanks!

[rowanborder]

I can confirm this issue is still present on L4T 36.4.4, Jetpack 6.2.1 and ISAAC ROS 3.2.

[vixp24]

@sgillen Do you have an update on the status of this bug? Is it resolved in Isaac ROS 4.0?

[sgillen]

Unfortunately I still don't have a specific fix for this issue, but please try with isaac ROS 4.0 if you are able.

Moving this back to image_proc as it appears to not be an issue with the argus node.