`ComposableNodeContainer` crashes when unloading `NitrosNode` with `ros2 component unload`.

Open Tuntenfisch opened this issue 8 months ago • 1 comments

Hello,

I need to dynamically load and unload NitrosNode instances in my multi-agent software stack. Specifically, I'm trying to load/unload nvidia::isaac_ros::h264_decoder::DecoderNode instances to decode compressed H.264 image data shared by other agents based on the presence/absence of these agent during runtime. To achieve this, I'm leveraging the ros2 component load/unload service interface. I noticed that when I unload an nvidia::isaac_ros::h264_decoder::DecoderNode the entire container crashes.

MWE:

example.launch.py

import launch
from launch_ros.actions import ComposableNodeContainer
from launch_ros.descriptions import ComposableNode


def generate_launch_description():
    example_container = ComposableNodeContainer(
        package="rclcpp_components",
        executable="component_container",
        name="example_container",
        namespace="example_namespace",
        composable_node_descriptions=(
            ComposableNode(
                package="isaac_ros_h264_decoder",
                plugin="nvidia::isaac_ros::h264_decoder::DecoderNode",
            ),
            ComposableNode(
                package="demo_nodes_cpp",
                plugin="demo_nodes_cpp::Talker",
            ),
        ),
        output="screen",
    )

    return launch.LaunchDescription([example_container])

When I launch the above launch file and subsequently try to unload the nvidia::isaac_ros::h264_decoder::DecoderNode with ros2 component unload /example_namespace/example_container 1, I get the following console output:

console output for single threaded component container

constantin@Laptop-CB:/workspaces/experimental_area/colcon_ws$ ros2 launch sfg_experimental_area example.launch.py 
[INFO] [launch]: All log files can be found below /home/constantin/.ros/log/2025-04-24-10-40-43-922251-Laptop-CB-78854
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container-1]: process started with pid [78866]
[component_container-1] [INFO] [1745484044.214638260] [example_namespace.example_container]: Load Library: /opt/ros/humble/lib/libdecoder_node.so
[component_container-1] [INFO] [1745484044.224797029] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::h264_decoder::DecoderNode>
[component_container-1] [INFO] [1745484044.224816699] [example_namespace.example_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::h264_decoder::DecoderNode>
[component_container-1] [INFO] [1745484044.370219755] [NitrosNode]: [NitrosNode] Initializing NitrosNode
[component_container-1] [INFO] [1745484044.370361594] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/std/libgxf_std.so
[component_container-1] [INFO] [1745484044.370875998] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_gxf_helpers.so
[component_container-1] [INFO] [1745484044.372162986] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_sight.so
[component_container-1] [INFO] [1745484044.373519589] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_atlas.so
[component_container-1] 2025-04-24 10:40:44.377 WARN  gxf/std/program.cpp@538: No GXF scheduler specified.
[component_container-1] [INFO] [1745484044.377278677] [NitrosNode]: [NitrosNode] Starting NitrosNode
[component_container-1] [INFO] [1745484044.378902133] [NitrosNode]: [NitrosNode] Loading extensions
[component_container-1] [INFO] [1745484044.378932143] [NitrosNode]: [NitrosContext] Loading extension: gxf/lib/multimedia/libgxf_multimedia.so
[component_container-1] [INFO] [1745484044.379987270] [NitrosNode]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_message_compositor.so
[component_container-1] [INFO] [1745484044.380152908] [NitrosNode]: [NitrosContext] Loading extension: gxf/lib/cuda/libgxf_cuda.so
[component_container-1] [INFO] [1745484044.381269680] [NitrosNode]: [NitrosContext] Loading extension: gxf/lib/serialization/libgxf_serialization.so
[component_container-1] [INFO] [1745484044.382710594] [NitrosNode]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_tensorops.so
[component_container-1] [INFO] [1745484044.384540365] [NitrosNode]: [NitrosContext] Loading extension: gxf/lib/codec/libgxf_video_decoder_extension.so
[component_container-1] [INFO] [1745484044.390871622] [NitrosNode]: [NitrosNode] Loading graph to the optimizer
[component_container-1] [INFO] [1745484044.391748899] [NitrosNode]: [NitrosNode] Running optimization
[component_container-1] [INFO] [1745484044.395374553] [NitrosNode]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container-1] [INFO] [1745484044.397174719] [NitrosNode]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/NitrosNode' in container '/example_namespace/example_container'
[component_container-1] [INFO] [1745484044.398981334] [example_namespace.example_container]: Load Library: /opt/ros/humble/lib/libtopics_library.so
[component_container-1] [INFO] [1745484044.399528577] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::ContentFilteringPublisher>
[component_container-1] [INFO] [1745484044.399535304] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::ContentFilteringSubscriber>
[component_container-1] [INFO] [1745484044.399538385] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::Listener>
[component_container-1] [INFO] [1745484044.399540740] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::ListenerBestEffort>
[component_container-1] [INFO] [1745484044.399542849] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::LoanedMessageTalker>
[component_container-1] [INFO] [1745484044.399544695] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::SerializedMessageListener>
[component_container-1] [INFO] [1745484044.399546564] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::SerializedMessageTalker>
[component_container-1] [INFO] [1745484044.399548463] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::Talker>
[component_container-1] [INFO] [1745484044.399550574] [example_namespace.example_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::Talker>
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/talker' in container '/example_namespace/example_container'
[component_container-1] [INFO] [1745484045.397528034] [NitrosNode]: [NitrosNode] Starting post negotiation setup
[component_container-1] [INFO] [1745484045.397577452] [NitrosNode]: [NitrosNode] Getting data format negotiation results
[component_container-1] [INFO] [1745484045.397589319] [NitrosNode]: [NitrosSubscriber] Negotiation ended with no results
[component_container-1] [INFO] [1745484045.397597067] [NitrosNode]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/image_compressed", data_format="nitros_compressed_image"
[component_container-1] [INFO] [1745484045.397653279] [NitrosNode]: [NitrosPublisher] Negotiation ended with no results
[component_container-1] [INFO] [1745484045.397661595] [NitrosNode]: [NitrosPublisher] Use only the compatible publisher: topic_name="/image_uncompressed", data_format="nitros_image_rgb8"
[component_container-1] [INFO] [1745484045.397771133] [NitrosNode]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container-1] [INFO] [1745484045.400185211] [NitrosNode]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/RHRHHSGGLK/RHRHHSGGLK.yaml"
[component_container-1] [INFO] [1745484045.400200070] [NitrosNode]: [DecoderNode] preLoadGraphCallback().
[component_container-1] [INFO] [1745484045.400210557] [NitrosNode]: [NitrosNode] Loading application
[component_container-1] [INFO] [1745484045.401571985] [NitrosNode]: [DecoderNode] postLoadGraphCallback().
[component_container-1] [INFO] [1745484045.401615510] [NitrosNode]: [NitrosNode] Initializing and running GXF graph
[component_container-1] [INFO] [1745484045.591538869] [NitrosNode]: [NitrosNode] Node was started
[component_container-1] [INFO] [1745484045.591755147] [talker]: Publishing: 'Hello World: 1'
[component_container-1] [INFO] [1745484046.400710812] [talker]: Publishing: 'Hello World: 2'
[component_container-1] [INFO] [1745484047.400775064] [talker]: Publishing: 'Hello World: 3'
[component_container-1] [INFO] [1745484048.400749492] [talker]: Publishing: 'Hello World: 4'
[component_container-1] [INFO] [1745484049.400815156] [talker]: Publishing: 'Hello World: 5'
[component_container-1] [INFO] [1745484050.400844608] [talker]: Publishing: 'Hello World: 6'
[component_container-1] [INFO] [1745484051.400798813] [talker]: Publishing: 'Hello World: 7'
[component_container-1] [INFO] [1745484052.400906021] [talker]: Publishing: 'Hello World: 8'
[component_container-1] [INFO] [1745484053.400829905] [talker]: Publishing: 'Hello World: 9'
[component_container-1] [INFO] [1745484054.400758481] [talker]: Publishing: 'Hello World: 10'
[component_container-1] [INFO] [1745484055.400899620] [talker]: Publishing: 'Hello World: 11'
[component_container-1] [INFO] [1745484056.400776694] [talker]: Publishing: 'Hello World: 12'
[component_container-1] [INFO] [1745484057.400883161] [talker]: Publishing: 'Hello World: 13'
[component_container-1] [INFO] [1745484058.400916632] [talker]: Publishing: 'Hello World: 14'
[component_container-1] [INFO] [1745484059.400865788] [talker]: Publishing: 'Hello World: 15'
[component_container-1] [INFO] [1745484060.400911203] [talker]: Publishing: 'Hello World: 16'
[component_container-1] [INFO] [1745484069.972032900] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::ContentFilteringPublisher>
[component_container-1] [INFO] [1745484069.972062742] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::ContentFilteringSubscriber>
[component_container-1] [INFO] [1745484069.972065335] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::Listener>
[component_container-1] [INFO] [1745484069.972067504] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::ListenerBestEffort>
[component_container-1] [INFO] [1745484069.972069510] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::LoanedMessageTalker>
[component_container-1] [INFO] [1745484069.972071502] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::SerializedMessageListener>
[component_container-1] [INFO] [1745484069.972074091] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::SerializedMessageTalker>
[component_container-1] [INFO] [1745484069.972076263] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::Talker>
[component_container-1] [INFO] [1745484069.972078209] [example_namespace.example_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::Talker>
[component_container-1] [INFO] [1745484070.973251870] [talker]: Publishing: 'Hello World: 1'
[component_container-1] [INFO] [1745484071.973378935] [talker]: Publishing: 'Hello World: 2'
[component_container-1] [INFO] [1745484072.973263418] [talker]: Publishing: 'Hello World: 3'
[component_container-1] [INFO] [1745484073.973349295] [talker]: Publishing: 'Hello World: 4'
[component_container-1] [INFO] [1745484074.749795506] [NitrosNode]: [NitrosNode] Terminating the running application
[component_container-1] [INFO] [1745484074.749818768] [NitrosNode]: [NitrosContext] Interrupting GXF...
[component_container-1] [INFO] [1745484074.750057681] [NitrosNode]: [NitrosContext] Waiting on GXF...
[component_container-1] [INFO] [1745484074.750063420] [NitrosNode]: [NitrosContext] Deinitializing...
[component_container-1] |==================================================================================================================================================================|
[component_container-1] |                                           Job Statistics Report (regular)                                                                                        |
[component_container-1] |==================================================================================================================================================================|
[component_container-1] | Name                                               |   Count | Time (Median - 90% - Max) [ms] | Load (%) | Exec(ms) | Variation (Median - 90% - Max) [ns]        |
[component_container-1] |------------------------------------------------------------------------------------------------------------------------------------------------------------------|
[component_container-1] |==================================================================================================================================================================|
[component_container-1] |==================================================================================================================================================================|
[component_container-1] |                                           Entity Statistics Report (regular)                                                                                     |
[component_container-1] |==================================================================================================================================================================|
[component_container-1] | Entity Name             | Entity State             |   Count | Time (Median - 90% - Max) [ms]                                                                    |
[component_container-1] |------------------------------------------------------------------------------------------------------------------------------------------------------------------|
[component_container-1] |==================================================================================================================================================================|
[component_container-1] [INFO] [1745484074.752878006] [NitrosNode]: [NitrosContext] Destroying context
[component_container-1] [INFO] [1745484074.752920305] [NitrosNode]: [NitrosNode] Application termination done
[component_container-1] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
[component_container-1]   what():  failed to add guard condition to wait set: guard condition implementation is invalid, at ./src/rcl/guard_condition.c:172, at ./src/rcl/wait.c:460
[ERROR] [component_container-1]: process has died [pid 78866, exit code -6, cmd '/opt/ros/humble/lib/rclcpp_components/component_container --ros-args -r __node:=example_container -r __ns:=/example_namespace'].

When, instead of a single threaded container I use a multi threaded container (i.e. component_container_mt), and repeat the launching/unloading procedure, I get another crash with the following output:

console output for multi threaded component container

constantin@Laptop-CB:/workspaces/experimental_area/colcon_ws$ ros2 launch sfg_experimental_area example.launch.py 
[INFO] [launch]: All log files can be found below /home/constantin/.ros/log/2025-04-24-10-51-35-987689-Laptop-CB-92872
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container_mt-1]: process started with pid [92884]
[component_container_mt-1] [INFO] [1745484696.280849121] [example_namespace.example_container]: Load Library: /opt/ros/humble/lib/libdecoder_node.so
[component_container_mt-1] [INFO] [1745484696.290371913] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::h264_decoder::DecoderNode>
[component_container_mt-1] [INFO] [1745484696.290388219] [example_namespace.example_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::h264_decoder::DecoderNode>
[component_container_mt-1] [INFO] [1745484696.449078895] [NitrosNode]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1745484696.449223289] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/std/libgxf_std.so
[component_container_mt-1] [INFO] [1745484696.449782402] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_gxf_helpers.so
[component_container_mt-1] [INFO] [1745484696.450943426] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_sight.so
[component_container_mt-1] [INFO] [1745484696.452376245] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_atlas.so
[component_container_mt-1] 2025-04-24 10:51:36.456 WARN  gxf/std/program.cpp@538: No GXF scheduler specified.
[component_container_mt-1] [INFO] [1745484696.456112915] [NitrosNode]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1745484696.457662526] [NitrosNode]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1745484696.457702455] [NitrosNode]: [NitrosContext] Loading extension: gxf/lib/multimedia/libgxf_multimedia.so
[component_container_mt-1] [INFO] [1745484696.458652599] [NitrosNode]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_message_compositor.so
[component_container_mt-1] [INFO] [1745484696.458816529] [NitrosNode]: [NitrosContext] Loading extension: gxf/lib/cuda/libgxf_cuda.so
[component_container_mt-1] [INFO] [1745484696.459877667] [NitrosNode]: [NitrosContext] Loading extension: gxf/lib/serialization/libgxf_serialization.so
[component_container_mt-1] [INFO] [1745484696.461244044] [NitrosNode]: [NitrosContext] Loading extension: gxf/lib/libgxf_isaac_tensorops.so
[component_container_mt-1] [INFO] [1745484696.462930886] [NitrosNode]: [NitrosContext] Loading extension: gxf/lib/codec/libgxf_video_decoder_extension.so
[component_container_mt-1] [INFO] [1745484696.472627808] [NitrosNode]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1745484696.473507964] [NitrosNode]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1745484696.477078385] [NitrosNode]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1745484696.479096932] [NitrosNode]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/NitrosNode' in container '/example_namespace/example_container'
[component_container_mt-1] [INFO] [1745484696.480662664] [example_namespace.example_container]: Load Library: /opt/ros/humble/lib/libtopics_library.so
[component_container_mt-1] [INFO] [1745484696.481424248] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::ContentFilteringPublisher>
[component_container_mt-1] [INFO] [1745484696.481438333] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::ContentFilteringSubscriber>
[component_container_mt-1] [INFO] [1745484696.481456790] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::Listener>
[component_container_mt-1] [INFO] [1745484696.481470123] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::ListenerBestEffort>
[component_container_mt-1] [INFO] [1745484696.481474992] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::LoanedMessageTalker>
[component_container_mt-1] [INFO] [1745484696.481477134] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::SerializedMessageListener>
[component_container_mt-1] [INFO] [1745484696.481479144] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::SerializedMessageTalker>
[component_container_mt-1] [INFO] [1745484696.481481307] [example_namespace.example_container]: Found class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::Talker>
[component_container_mt-1] [INFO] [1745484696.481483478] [example_namespace.example_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<demo_nodes_cpp::Talker>
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/talker' in container '/example_namespace/example_container'
[component_container_mt-1] [INFO] [1745484697.479482374] [NitrosNode]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1745484697.479530673] [NitrosNode]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1745484697.479543575] [NitrosNode]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1745484697.479550165] [NitrosNode]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/image_compressed", data_format="nitros_compressed_image"
[component_container_mt-1] [INFO] [1745484697.479602582] [NitrosNode]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1745484697.479611692] [NitrosNode]: [NitrosPublisher] Use only the compatible publisher: topic_name="/image_uncompressed", data_format="nitros_image_rgb8"
[component_container_mt-1] [INFO] [1745484697.479721849] [NitrosNode]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1745484697.481641744] [NitrosNode]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/GDDSXCCCQX/GDDSXCCCQX.yaml"
[component_container_mt-1] [INFO] [1745484697.481679564] [NitrosNode]: [DecoderNode] preLoadGraphCallback().
[component_container_mt-1] [INFO] [1745484697.481682235] [NitrosNode]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1745484697.482504006] [NitrosNode]: [DecoderNode] postLoadGraphCallback().
[component_container_mt-1] [INFO] [1745484697.482511712] [NitrosNode]: [NitrosNode] Initializing and running GXF graph
[component_container_mt-1] [INFO] [1745484697.483078543] [talker]: Publishing: 'Hello World: 1'
[component_container_mt-1] [INFO] [1745484697.675944291] [NitrosNode]: [NitrosNode] Node was started
[component_container_mt-1] [INFO] [1745484698.483150433] [talker]: Publishing: 'Hello World: 2'
[component_container_mt-1] [INFO] [1745484699.483088456] [talker]: Publishing: 'Hello World: 3'
[component_container_mt-1] [INFO] [1745484700.483216647] [talker]: Publishing: 'Hello World: 4'
[component_container_mt-1] [INFO] [1745484701.483294328] [talker]: Publishing: 'Hello World: 5'
[component_container_mt-1] [INFO] [1745484702.483075165] [talker]: Publishing: 'Hello World: 6'
[component_container_mt-1] [INFO] [1745484703.483141320] [talker]: Publishing: 'Hello World: 7'
[component_container_mt-1] [INFO] [1745484704.483096430] [talker]: Publishing: 'Hello World: 8'
[component_container_mt-1] [INFO] [1745484705.483185364] [talker]: Publishing: 'Hello World: 9'
[component_container_mt-1] [INFO] [1745484706.483289123] [talker]: Publishing: 'Hello World: 10'
[component_container_mt-1] [INFO] [1745484707.483297641] [talker]: Publishing: 'Hello World: 11'
[component_container_mt-1] [INFO] [1745484708.483148027] [talker]: Publishing: 'Hello World: 12'
[component_container_mt-1] [INFO] [1745484709.483219859] [talker]: Publishing: 'Hello World: 13'
[component_container_mt-1] [INFO] [1745484710.483286500] [talker]: Publishing: 'Hello World: 14'
[component_container_mt-1] [INFO] [1745484711.157555502] [NitrosNode]: [NitrosNode] Terminating the running application
[component_container_mt-1] [INFO] [1745484711.157609201] [NitrosNode]: [NitrosContext] Interrupting GXF...
[component_container_mt-1] [INFO] [1745484711.157880500] [NitrosNode]: [NitrosContext] Waiting on GXF...
[component_container_mt-1] [INFO] [1745484711.157918837] [NitrosNode]: [NitrosContext] Deinitializing...
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] |                                           Job Statistics Report (regular)                                                                                        |
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] | Name                                               |   Count | Time (Median - 90% - Max) [ms] | Load (%) | Exec(ms) | Variation (Median - 90% - Max) [ns]        |
[component_container_mt-1] |------------------------------------------------------------------------------------------------------------------------------------------------------------------|
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] |                                           Entity Statistics Report (regular)                                                                                     |
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] | Entity Name             | Entity State             |   Count | Time (Median - 90% - Max) [ms]                                                                    |
[component_container_mt-1] |------------------------------------------------------------------------------------------------------------------------------------------------------------------|
[component_container_mt-1] |==================================================================================================================================================================|
[component_container_mt-1] [INFO] [1745484711.163499756] [NitrosNode]: [NitrosContext] Destroying context
[component_container_mt-1] [INFO] [1745484711.163603151] [NitrosNode]: [NitrosNode] Application termination done
[ERROR] [component_container_mt-1]: process has died [pid 92884, exit code -11, cmd '/opt/ros/humble/lib/rclcpp_components/component_container_mt --ros-args -r __node:=example_container -r __ns:=/example_namespace'].

Note that, while unloading nvidia::isaac_ros::h264_decoder::DecoderNode always results in a crash, unloading demo_nodes_cpp::Talker (which I included for testing purposes) works without issue. Also loading additional nvidia::isaac_ros::h264_decoder::DecoderNode instances works.

Apr 24 '25 09:04 Tuntenfisch

Hello,

I need to dynamically load and unload NitrosNode instances in my multi-agent software stack. Specifically, I'm trying to load/unload nvidia::isaac_ros::h264_decoder::DecoderNode instances to decode compressed H.264 image data shared by other agents based on the presence/absence of these agent during runtime. To achieve this, I'm leveraging the ros2 component load/unload service interface. I noticed that when I unload an nvidia::isaac_ros::h264_decoder::DecoderNode the entire container crashes.

MWE: example.launch.py

When I launch the above launch file and subsequently try to unload the nvidia::isaac_ros::h264_decoder::DecoderNode with ros2 component unload /example_namespace/example_container 1, I get the following console output: console output for single threaded component container

When, instead of a single threaded container I use a multi threaded container (i.e. component_container_mt), and repeat the launching/unloading procedure, I get another crash with the following output: console output for multi threaded component container

Note that, while unloading nvidia::isaac_ros::h264_decoder::DecoderNode always results in a crash, unloading demo_nodes_cpp::Talker (which I included for testing purposes) works without issue. Also loading additional nvidia::isaac_ros::h264_decoder::DecoderNode instances works.

I've attached gdb to the MWE described above:

debugger output for single threaded component container

constantin@Laptop-CB:/workspaces$ sudo gdb attach 19338
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04.2) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
attach: No such file or directory.
Attaching to process 19338
[New LWP 19340]
[New LWP 19341]
[New LWP 19342]
[New LWP 19343]
[New LWP 19344]
[New LWP 19345]
[New LWP 19346]
[New LWP 19347]
[New LWP 19348]
[New LWP 19353]
[New LWP 19354]
[New LWP 19355]
[New LWP 19469]
[New LWP 19470]
[New LWP 19499]
[New LWP 19500]
[New LWP 19501]
[New LWP 19502]
[New LWP 19503]
[New LWP 19504]
[New LWP 19505]
[New LWP 19506]
[New LWP 19507]
[New LWP 19508]
[New LWP 19509]
[New LWP 19510]
[New LWP 19511]
[New LWP 19512]
[New LWP 19513]
[New LWP 19514]
[New LWP 19515]
[New LWP 19516]
[New LWP 19517]
[New LWP 19518]
[New LWP 19519]
[New LWP 19520]
[New LWP 19521]
[New LWP 19522]
[New LWP 19523]
[New LWP 19524]
[New LWP 19525]
[New LWP 19526]
[New LWP 19527]
[New LWP 19528]
[New LWP 19529]
[New LWP 19530]
[New LWP 19531]
[New LWP 19549]
[New LWP 19550]
[New LWP 19551]
[New LWP 19552]
[New LWP 19553]
[New LWP 19554]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
__futex_abstimed_wait_common64 (private=1606620704, cancel=true, abstime=0x7fff07c9b100, op=137, expected=0, futex_word=0x596a5f275a80) at ./nptl/futex-internal.c:57
57      ./nptl/futex-internal.c: No such file or directory.
(gdb) continue
Continuing.
[Thread 0x773e4fe00640 (LWP 19553) exited]
[Thread 0x773e59000640 (LWP 19552) exited]
[Thread 0x773e59a00640 (LWP 19551) exited]
[Thread 0x773e5a400640 (LWP 19550) exited]
[Thread 0x773e5ae00640 (LWP 19549) exited]
[Thread 0x773e64c00640 (LWP 19531) exited]

Thread 1 "component_conta" received signal SIGABRT, Aborted.
__pthread_kill_implementation (no_tid=0, signo=6, threadid=131112917155648) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=131112917155648)
    at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=131112917155648) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=131112917155648, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x0000773f1b0a3476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x0000773f1b0897f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x0000773f1b34eb9e in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x0000773f1b35a20c in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x0000773f1b35a277 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x0000773f1b35a1fe in std::rethrow_exception(std::__exception_ptr::exception_ptr) ()
   from /lib/x86_64-linux-gnu/libstdc++.so.6
#9  0x0000773f1b5f5139 in rclcpp::exceptions::throw_from_rcl_error(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rcutils_error_state_s const*, void (*)()) ()
   from /opt/ros/humble/lib/librclcpp.so
#10 0x0000773f1b5f8fac in rclcpp::detail::add_guard_condition_to_rcl_wait_set(rcl_wait_set_s&, rclcpp::GuardCondition const&) () from /opt/ros/humble/lib/librclcpp.so
#11 0x0000773f1b60b8e7 in ?? () from /opt/ros/humble/lib/librclcpp.so
#12 0x0000773f1b5f8795 in rclcpp::Executor::wait_for_work(std::chrono::duration<long, std::ratio<1l, 1000000000l> >) () from /opt/ros/humble/lib/librclcpp.so
#13 0x0000773f1b5fb4f3 in rclcpp::Executor::get_next_executable(rclcpp::AnyExecutable&, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) () from /opt/ros/humble/lib/librclcpp.so
#14 0x0000773f1b602aa1 in rclcpp::executors::SingleThreadedExecutor::spin() ()
   from /opt/ros/humble/lib/librclcpp.so

debugger output for multi threaded component container

constantin@Laptop-CB:/workspaces$ sudo gdb attach 1685
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04.2) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
attach: No such file or directory.
Attaching to process 1685
[New LWP 1687]
[New LWP 1688]
[New LWP 1689]
[New LWP 1690]
[New LWP 1691]
[New LWP 1692]
[New LWP 1693]
[New LWP 1694]
[New LWP 1695]
[New LWP 1696]
[New LWP 1697]
[New LWP 1698]
[New LWP 1699]
[New LWP 1700]
[New LWP 1701]
[New LWP 1702]
[New LWP 1703]
[New LWP 1704]
[New LWP 1705]
[New LWP 1706]
[New LWP 1707]
[New LWP 1708]
[New LWP 1709]
[New LWP 1710]
[New LWP 1711]
[New LWP 1712]
[New LWP 1713]
[New LWP 1714]
[New LWP 1715]
[New LWP 1716]
[New LWP 1717]
[New LWP 1718]
[New LWP 1719]
[New LWP 1720]
[New LWP 1721]
[New LWP 1722]
[New LWP 1723]
[New LWP 1724]
[New LWP 1725]
[New LWP 1726]
[New LWP 1763]
[New LWP 1791]
[New LWP 1792]
[New LWP 1983]
[New LWP 1984]
[New LWP 2013]
[New LWP 2014]
[New LWP 2015]
[New LWP 2016]
[New LWP 2017]
[New LWP 2018]
[New LWP 2019]
[New LWP 2020]
[New LWP 2021]
[New LWP 2022]
[New LWP 2023]
[New LWP 2024]
[New LWP 2025]
[New LWP 2026]
[New LWP 2027]
[New LWP 2028]
[New LWP 2029]
[New LWP 2030]
[New LWP 2031]
[New LWP 2032]
[New LWP 2033]
[New LWP 2034]
[New LWP 2035]
[New LWP 2036]
[New LWP 2037]
[New LWP 2038]
[New LWP 2039]
[New LWP 2040]
[New LWP 2041]
[New LWP 2042]
[New LWP 2043]
[New LWP 2044]
[New LWP 2045]
[New LWP 2063]
[New LWP 2064]
[New LWP 2065]
[New LWP 2066]
[New LWP 2067]
[New LWP 2068]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
futex_wait (private=0, expected=2, futex_word=0x60e0eb046470) at ../sysdeps/nptl/futex-internal.h:146
146     ../sysdeps/nptl/futex-internal.h: No such file or directory.
(gdb) continue
Continuing.
[Thread 0x7c12b7400640 (LWP 2067) exited]
[Thread 0x7c12b7e00640 (LWP 2066) exited]
[Thread 0x7c12c0a00640 (LWP 2065) exited]
[Thread 0x7c12c1400640 (LWP 2064) exited]
[Thread 0x7c12c1e00640 (LWP 2063) exited]
[Thread 0x7c12c3e00640 (LWP 2045) exited]

Thread 32 "component_conta" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7c1352400640 (LWP 1717)]
0x00007c137ae2be3f in rmw_fastrtps_shared_cpp::__rmw_wait(char const*, rmw_subscriptions_s*, rmw_guard_conditions_s*, rmw_services_s*, rmw_clients_s*, rmw_events_s*, rmw_wait_set_s*, rmw_time_s const*) () from /opt/ros/humble/lib/librmw_fastrtps_shared_cpp.so
(gdb) bt
#0  0x00007c137ae2be3f in rmw_fastrtps_shared_cpp::__rmw_wait(char const*, rmw_subscriptions_s*, rmw_guard_conditions_s*, rmw_services_s*, rmw_clients_s*, rmw_events_s*, rmw_wait_set_s*, rmw_time_s const*) ()
   from /opt/ros/humble/lib/librmw_fastrtps_shared_cpp.so
#1  0x00007c137ae83887 in rmw_wait () from /opt/ros/humble/lib/librmw_fastrtps_cpp.so
#2  0x00007c137b12c848 in rcl_wait () from /opt/ros/humble/lib/librcl.so
#3  0x00007c137b6f77bc in rclcpp::Executor::wait_for_work(std::chrono::duration<long, std::ratio<1l, 1000000000l> >) () from /opt/ros/humble/lib/librclcpp.so
#4  0x00007c137b6fa4f3 in rclcpp::Executor::get_next_executable(rclcpp::AnyExecutable&, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) () from /opt/ros/humble/lib/librclcpp.so
#5  0x00007c137b701452 in rclcpp::executors::MultiThreadedExecutor::run(unsigned long) ()
   from /opt/ros/humble/lib/librclcpp.so
#6  0x00007c137b488253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x00007c137b1f5ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#8  0x00007c137b287850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

I suspect the issue is a combination between erroneous cleanup of the NitrosNode and its created publishers/subscribers/..., and a resulting race condition.

Apr 24 '25 09:04 Tuntenfisch