isaac_ros_object_detection icon indicating copy to clipboard operation
isaac_ros_object_detection copied to clipboard

Published 20 hours ago • verified publisher icon NVIDIA-ISAAC-ROS

Cuda bindings mismatch error using custom trained model

Open immaixq opened this issue 1 year ago • 18 comments

Description: Encounter error using yolov8_tensor_rt.launch.py and isaac_ros_yolov8_visualize.launch.py. The scripts are run on jetson orin nano. The custom trained model is exported to onnx as suggested by documentation using ultralytics, I have also tried exporting using trtexec but still face the same error.

Screenshot from 2023-11-27 09-58-40

Thank you for your help.

immaixq avatar Nov 27 '23 03:11 immaixq

Hi @immaixq Have you figured this out?

I am also trying to use a custom yolov8 model, but failed to do so. I also converted the .pt file to onnx and the to an engine file, but I am not sure If I need to change any of the arguments that are defined in the following command

   ros2 launch isaac_ros_yolov8 isaac_ros_yolov8_visualize.launch.py model_file_path:=/tmp/yolov8s.onnx engine_file_path:=/tmp/yolov8s.plan input_binding_names:=['images'] output_binding_names:=['output0'] network_image_width:=640 network_image_height:=640 force_engine_update:=False image_mean:=[0.0,0.0,0.0] image_stddev:=[1.0,1.0,1.0] input_image_width:=640 input_image_height:=640 confidence_threshold:=0.25 nms_threshold:=0.45

mzahana avatar Nov 27 '23 13:11 mzahana

I suspect that the issue is in the hardcoded number of classes https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_object_detection/blob/b43ecaed3e1be3864a5f872e105aa3bbecd05a41/isaac_ros_yolov8/src/yolov8_decoder_node.cpp#L76

If your custom model has different number of classes then it will break here https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_object_detection/blob/b43ecaed3e1be3864a5f872e105aa3bbecd05a41/isaac_ros_yolov8/src/yolov8_decoder_node.cpp#L92

I suggest that you visualize your ONNX model using netron, and note the input/output names and the input/output dimensions.

mzahana avatar Nov 27 '23 13:11 mzahana

Hi @immaixq,

The problem you posted seems to originate from a parameter mismatch. We have parameters called input_binding_names and output_binding_names, as well as input_tensor_names and output_tensor_names. Please see here for where they are located in the launch file. I believe that these need to be updated accordingly to match what your model expects.

The input/output binding names should be noted from the ONNX model as @mzahana said. The input tensor names can be kept the same. The output tensor name I believe should be output_tensor based on here.

It also appears that your model outputs a 3 tensors, instead of 2 as we expect. Having additional information about your network would likely be useful.

kajananchinniahNV avatar Nov 27 '23 17:11 kajananchinniahNV

Hi @mzahana,

To respond to your issue separately, hopefully my first response is helpful for your first issue.

You're also right that the number of classes expected by the decoder node is hard coded, which will be problematic later. Currently, the issue seems to originate from the TensorRT node, and not the decoder.

kajananchinniahNV avatar Nov 27 '23 17:11 kajananchinniahNV

@kajananchinniahNV thanks for your response.

My problem was solved after I changed the hard-coded number of classes according to my custom model.

I suggest to make it a parameter.

Thanks.

mzahana avatar Nov 27 '23 21:11 mzahana

Thanks for finding a fix to the issue! I'll remove the needs info label and keep the bug label.

We'll be working on a patch to this issue soon. The solution you recommend sounds reasonable to me.

@immaixq could you confirm that this also solves it for you?

kajananchinniahNV avatar Nov 27 '23 21:11 kajananchinniahNV

@kajananchinniahNV thank you.

Actually, to make the fix more complete, the output dimension of the model could be a parameter. This way, the decoder can adapt to custom models with output tensor dimensions(not only the number of classes) that is different from the default one by Yolov8 [1, 84, 8400].

Thanks.

mzahana avatar Nov 27 '23 21:11 mzahana

@mzahana Thank you for your help and advice, I have gained some insights after looking into netron again. I did changed the number of classes in the script and I think the error is caused by the number of outputs my model gives, which consists of output0 and output1. I managed to fix the mismatch error after changing output_tensor_names and output_binding_names to [output0, output1]. However, another error occurred which I'm not sure of what could be the cause of it.

@kajananchinniahNV Thank you for your help, I have changed the parameters as mentioned above and managed to fix the mismatch error. Unfortunately another error occurred. The process has died without additional log on its cause. it'd be great if you could share some insights or advice on how to troubleshoot the cause of it.

Really appreciate both of your help, thank you!

Screenshot from 2023-11-28 11-39-54

immaixq avatar Nov 28 '23 04:11 immaixq

I'll be reopening this until we have a solution merged in the repository for tracking purposes.

We'll close it once a fix is in place.

kajananchinniahNV avatar Nov 28 '23 04:11 kajananchinniahNV

Hello, i'm coming from another issue post, that has the same problem. I did check the neutron model structure, but it did not differ from the original yolov8s model, except from the output size that goes from [1,84,8400] to [1,5,8400]. For this reason I added a parameter into the yolov8_decoder_node.hpp and yolov8_decoder_node.cpp that can be modified from the launch file, in order to take in account the class number change. But still everything crashes after launch, the problem seems to be inside the tensor_rt node. here it is the output of neutron: image

while here it is the run of the tesor_rt_node without any encoder/decoder: cd /workspaces/isaac_ros-dev && ros2 launch isaac_ros_yolov8 isaac_ros_yolov8_visualize.launch.py model_file_path:=/tmp/s_best.onnx engine_file_path:=/tmp/s_best.plan network_image_width:=640 network_image_height:=640 force_engine_update:=False image_mean:=[0.0,0.0,0.0] image_stddev:=[1.0,1.0,1.0] input_image_width:=640 input_image_height:=640 confidence_threshold:=0.25 nms_threshold:=0.45 num_classes:=1 [INFO] [launch]: All log files can be found below /home/admin/.ros/log/2023-11-30-19-34-51-284081-Egidio-317485 [INFO] [launch]: Default logging verbosity is set to INFO [INFO] [component_container_mt-1]: process started with pid [317496] [INFO] [isaac_ros_yolov8_visualizer.py-2]: process started with pid [317498] [INFO] [rqt_image_view-3]: process started with pid [317500] [component_container_mt-1] [INFO] [1701369291.699265543] [tensor_rt_container]: Load Library: /opt/ros/humble/lib/libdnn_image_encoder_node.so [component_container_mt-1] [INFO] [1701369291.709586471] [tensor_rt_container]: Found class: rclcpp_components::NodeFactoryTemplatenvidia::isaac_ros::dnn_inference::DnnImageEncoderNode [component_container_mt-1] [INFO] [1701369291.709630123] [tensor_rt_container]: Instantiate class: rclcpp_components::NodeFactoryTemplatenvidia::isaac_ros::dnn_inference::DnnImageEncoderNode [component_container_mt-1] [INFO] [1701369291.712210486] [NitrosContext]: [NitrosContext] Creating a new shared context [component_container_mt-1] [INFO] [1701369291.712677614] [dnn_image_encoder]: [NitrosNode] Initializing NitrosNode [component_container_mt-1] [INFO] [1701369291.713071269] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/std/libgxf_std.so [component_container_mt-1] [INFO] [1701369291.717046746] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_gxf_helpers.so [component_container_mt-1] [INFO] [1701369291.719117256] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_sight.so [component_container_mt-1] [INFO] [1701369291.721440821] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_atlas.so [component_container_mt-1] [INFO] [1701369291.724068618] [NitrosContext]: [NitrosContext] Loading application: '/opt/ros/humble/share/isaac_ros_nitros/config/type_adapter_nitros_context_graph.yaml' [component_container_mt-1] [INFO] [1701369291.724616426] [NitrosContext]: [NitrosContext] Initializing application... [component_container_mt-1] [INFO] [1701369291.727872325] [NitrosContext]: [NitrosContext] Running application... [component_container_mt-1] 2023-11-30 19:34:51.727 WARN gxf/std/program.cpp@514: No system specified. Nothing to do [component_container_mt-1] [INFO] [1701369291.728550577] [dnn_image_encoder]: [NitrosNode] Starting NitrosNode [component_container_mt-1] [INFO] [1701369291.728565274] [dnn_image_encoder]: [NitrosNode] Loading built-in preset extension specs [component_container_mt-1] [INFO] [1701369291.729633893] [dnn_image_encoder]: [NitrosNode] Loading built-in extension specs [component_container_mt-1] [INFO] [1701369291.729643912] [dnn_image_encoder]: [NitrosNode] Loading preset extension specs [component_container_mt-1] [INFO] [1701369291.730784416] [dnn_image_encoder]: [NitrosNode] Loading extension specs [component_container_mt-1] [INFO] [1701369291.730793663] [dnn_image_encoder]: [NitrosNode] Loading generator rules [component_container_mt-1] [INFO] [1701369291.730977303] [dnn_image_encoder]: [NitrosNode] Loading extensions [component_container_mt-1] [INFO] [1701369291.731084571] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/lib/libgxf_message_compositor.so [component_container_mt-1] [INFO] [1701369291.731892282] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/lib/cuda/libgxf_cuda.so [component_container_mt-1] [INFO] [1701369291.732606752] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/lib/serialization/libgxf_serialization.so [component_container_mt-1] [INFO] [1701369291.735061868] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/lib/image_proc/libgxf_tensorops.so [component_container_mt-1] [INFO] [1701369291.738366425] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/multimedia/libgxf_multimedia.so [component_container_mt-1] [INFO] [1701369291.739427020] [dnn_image_encoder]: [NitrosNode] Loading graph to the optimizer [component_container_mt-1] [INFO] [1701369291.742600550] [dnn_image_encoder]: [NitrosNode] Running optimization [component_container_mt-1] [INFO] [1701369291.829503900] [dnn_image_encoder]: [NitrosNode] Obtaining graph IO group info from the optimizer [component_container_mt-1] [INFO] [1701369291.831678458] [dnn_image_encoder]: [NitrosNode] Creating negotiated publishers/subscribers [component_container_mt-1] [INFO] [1701369291.839007725] [dnn_image_encoder]: [NitrosNode] Starting negotiation... [INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/dnn_image_encoder' in container '/tensor_rt_container' [component_container_mt-1] [INFO] [1701369291.841603810] [tensor_rt_container]: Load Library: /opt/ros/humble/lib/libtensor_rt_node.so [component_container_mt-1] [INFO] [1701369291.874405223] [tensor_rt_container]: Found class: rclcpp_components::NodeFactoryTemplatenvidia::isaac_ros::dnn_inference::TensorRTNode [component_container_mt-1] [INFO] [1701369291.874462346] [tensor_rt_container]: Instantiate class: rclcpp_components::NodeFactoryTemplatenvidia::isaac_ros::dnn_inference::TensorRTNode [component_container_mt-1] [INFO] [1701369291.877348066] [tensor_rt]: [NitrosNode] Initializing NitrosNode [component_container_mt-1] [INFO] [1701369291.877813663] [tensor_rt]: [NitrosNode] Starting NitrosNode [component_container_mt-1] [INFO] [1701369291.877821802] [tensor_rt]: [NitrosNode] Loading built-in preset extension specs [component_container_mt-1] [INFO] [1701369291.879300150] [tensor_rt]: [NitrosNode] Loading built-in extension specs [component_container_mt-1] [INFO] [1701369291.879320782] [tensor_rt]: [NitrosNode] Loading preset extension specs [component_container_mt-1] [INFO] [1701369291.879602509] [tensor_rt]: [NitrosNode] Loading extension specs [component_container_mt-1] [INFO] [1701369291.879607005] [tensor_rt]: [NitrosNode] Loading generator rules [component_container_mt-1] [INFO] [1701369291.879739664] [tensor_rt]: [NitrosNode] Loading extensions [component_container_mt-1] [INFO] [1701369291.880154519] [tensor_rt]: [NitrosContext] Loading extension: gxf/tensor_rt/libgxf_tensor_rt.so [component_container_mt-1] [INFO] [1701369291.881506225] [tensor_rt]: [NitrosNode] Loading graph to the optimizer [component_container_mt-1] [INFO] [1701369291.882449822] [tensor_rt]: [NitrosNode] Running optimization [component_container_mt-1] [INFO] [1701369291.890357829] [tensor_rt]: [NitrosNode] Obtaining graph IO group info from the optimizer [component_container_mt-1] [INFO] [1701369291.891196735] [tensor_rt]: [NitrosNode] Creating negotiated publishers/subscribers [component_container_mt-1] [INFO] [1701369291.891224289] [tensor_rt]: [NitrosPublisherSubscriberGroup] Pinning the component "inference/rx" (type="nvidia::gxf::DoubleBufferReceiver") to use its compatible format only: "nitros_tensor_list_nchw_rgb_f32" [component_container_mt-1] [INFO] [1701369291.891236891] [tensor_rt]: [NitrosPublisherSubscriberGroup] Pinning the component "sink/sink" (type="nvidia::isaac_ros::MessageRelay") to use its compatible format only: "nitros_tensor_list_nhwc_rgb_f32" [component_container_mt-1] [INFO] [1701369291.892752454] [tensor_rt]: [NitrosNode] Starting negotiation... [INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/tensor_rt' in container '/tensor_rt_container' [component_container_mt-1] [INFO] [1701369291.893821691] [tensor_rt_container]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_yolov8/lib/libyolov8_decoder_node.so [component_container_mt-1] [INFO] [1701369291.897479472] [tensor_rt_container]: Found class: rclcpp_components::NodeFactoryTemplatenvidia::isaac_ros::yolov8::YoloV8DecoderNode [component_container_mt-1] [INFO] [1701369291.897495971] [tensor_rt_container]: Instantiate class: rclcpp_components::NodeFactoryTemplatenvidia::isaac_ros::yolov8::YoloV8DecoderNode [component_container_mt-1] [INFO] [1701369291.900536323] [yolov8_decoder_node.ManagedNitrosSubscriber]: Starting Managed Nitros Subscriber [component_container_mt-1] [INFO] [1701369291.900599009] [tensor_rt]: Negotiating [component_container_mt-1] [INFO] [1701369291.900636307] [tensor_rt]: Could not negotiate [INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/yolov8_decoder_node' in container '/tensor_rt_container' [component_container_mt-1] [INFO] [1701369291.934111873] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369291.935284281] [dnn_image_encoder]: Negotiating [component_container_mt-1] [INFO] [1701369291.947272058] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369291.982223412] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.016896335] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.049667792] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.083270206] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.116083800] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.150248000] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.184162641] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.217759223] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.251089681] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.283518456] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.318538717] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.351548301] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.384940387] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.416969910] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.449342500] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.484723863] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.516197962] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.550623352] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.583731148] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.616523174] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.650641510] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.684008235] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.716650965] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.749464186] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.783434003] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.816425523] [dnn_image_encoder]: [NitrosSubscriber] Received a message but the application receiver's pointer is not yet set. [component_container_mt-1] [INFO] [1701369292.839439958] [dnn_image_encoder]: [NitrosNode] Starting post negotiation setup [component_container_mt-1] [INFO] [1701369292.839537394] [dnn_image_encoder]: [NitrosNode] Getting data format negotiation results [component_container_mt-1] [INFO] [1701369292.839562452] [dnn_image_encoder]: [NitrosPublisher] Use the negotiated data format: "nitros_tensor_list_nchw_rgb_f32" [component_container_mt-1] [INFO] [1701369292.839576597] [dnn_image_encoder]: [NitrosSubscriber] Negotiation ended with no results [component_container_mt-1] [INFO] [1701369292.839590433] [dnn_image_encoder]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/image", data_format="nitros_image_bgr8" [component_container_mt-1] [INFO] [1701369292.839900061] [dnn_image_encoder]: [NitrosNode] Exporting the final graph based on the negotiation results [component_container_mt-1] [INFO] [1701369292.880184565] [dnn_image_encoder]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/VLIIWSZCED/VLIIWSZCED.yaml" [component_container_mt-1] [INFO] [1701369292.880220173] [dnn_image_encoder]: [NitrosNode] Calling user's pre-load-graph callback [component_container_mt-1] [INFO] [1701369292.880227743] [dnn_image_encoder]: In DNN Image Encoder Node preLoadGraphCallback(). [component_container_mt-1] [INFO] [1701369292.880249907] [dnn_image_encoder]: [NitrosNode] Loading application [component_container_mt-1] [INFO] [1701369292.880254837] [dnn_image_encoder]: [NitrosContext] Loading application: '/tmp/isaac_ros_nitros/graphs/VLIIWSZCED/VLIIWSZCED.yaml' [component_container_mt-1] [INFO] [1701369292.882713726] [dnn_image_encoder]: [NitrosNode] Linking Nitros pub/sub to the loaded application [component_container_mt-1] [INFO] [1701369292.882774628] [dnn_image_encoder]: [NitrosNode] Calling user's post-load-graph callback [component_container_mt-1] [INFO] [1701369292.882778619] [dnn_image_encoder]: In DNN Image Encoder Node postLoadGraphCallback(). [component_container_mt-1] [INFO] [1701369292.882860822] [dnn_image_encoder]: [NitrosContext] Initializing application... [component_container_mt-1] [INFO] [1701369292.891619948] [dnn_image_encoder]: [NitrosContext] Running application... [component_container_mt-1] [INFO] [1701369292.891754417] [dnn_image_encoder]: [NitrosNode] Starting a heartbeat timer (eid=137) [component_container_mt-1] [INFO] [1701369292.892987384] [tensor_rt]: [NitrosNode] Starting post negotiation setup [component_container_mt-1] [INFO] [1701369292.893010583] [tensor_rt]: [NitrosNode] Getting data format negotiation results [component_container_mt-1] [INFO] [1701369292.893014784] [tensor_rt]: [NitrosPublisher] Negotiation ended with no results [component_container_mt-1] [INFO] [1701369292.893019335] [tensor_rt]: [NitrosPublisher] Use only the compatible publisher: topic_name="/tensor_sub", data_format="nitros_tensor_list_nhwc_rgb_f32" [component_container_mt-1] [INFO] [1701369292.893050205] [tensor_rt]: [NitrosSubscriber] Use the negotiated data format: "nitros_tensor_list_nchw_rgb_f32" [component_container_mt-1] [INFO] [1701369292.893151012] [tensor_rt]: [NitrosNode] Exporting the final graph based on the negotiation results [component_container_mt-1] [INFO] [1701369292.895767175] [tensor_rt]: [NitrosNode] Wrote the final top level YAML graph to "/tmp/isaac_ros_nitros/graphs/CVQLNEENCN/CVQLNEENCN.yaml" [component_container_mt-1] [INFO] [1701369292.895789827] [tensor_rt]: [NitrosNode] Calling user's pre-load-graph callback [component_container_mt-1] [INFO] [1701369292.895793439] [tensor_rt]: [NitrosNode] Loading application [component_container_mt-1] [INFO] [1701369292.895798445] [tensor_rt]: [NitrosContext] Loading application: '/tmp/isaac_ros_nitros/graphs/CVQLNEENCN/CVQLNEENCN.yaml' [component_container_mt-1] 2023-11-30 19:34:52.897 WARN gxf/std/yaml_file_loader.cpp@1058: Using unregistered parameter 'dummy_rx' in component ''. [component_container_mt-1] 2023-11-30 19:34:52.897 WARN gxf/std/yaml_file_loader.cpp@1058: Using unregistered parameter 'dev_id' in component 'stream'. [component_container_mt-1] [INFO] [1701369292.897544799] [tensor_rt]: [NitrosNode] Linking Nitros pub/sub to the loaded application [component_container_mt-1] [INFO] [1701369292.897606722] [tensor_rt]: [NitrosNode] Calling user's post-load-graph callback [component_container_mt-1] [INFO] [1701369292.897615558] [tensor_rt]: In TensorRTNode postLoadGraphCallback(). [component_container_mt-1] [ERROR] [1701369293.026835734] [TRT]: TRT ERROR: 3: [engine.cpp::getProfileObliviousBindingIndex::1401] Error Code 3: Internal Error (getTensorShape given invalid tensor name: ) [component_container_mt-1] [ERROR] [1701369293.034368876] [TRT]: TRT ERROR: 3: [engine.cpp::getProfileObliviousBindingIndex::1401] Error Code 3: Internal Error (getTensorDataType given invalid tensor name: ) [component_container_mt-1] [INFO] [1701369293.042799419] [tensor_rt]: Read tensor shape information from TRT Model Engine: /tmp/s_best.plan [component_container_mt-1] [INFO] [1701369293.042881374] [tensor_rt]: Tensors 4 bytes, num outputs 40 x tensors per output 1 = 40 blocks [component_container_mt-1] [INFO] [1701369293.042948886] [tensor_rt]: [NitrosContext] Initializing application... [component_container_mt-1] [INFO] [1701369293.046011715] [tensor_rt]: [NitrosContext] Running application... [component_container_mt-1] [INFO] [1701369293.046168062] [tensor_rt]: [NitrosNode] Starting a heartbeat timer (eid=254) [component_container_mt-1] 2023-11-30 19:34:53.109 ERROR /home/jenkins/agent/workspace/ly_debian-compatible-release-2.1/isaac_ros-dev/ros_ws/src/isaac_ros_dnn_inference/isaac_ros_tensor_rt/gxf/tensor_rt/tensor_rt_inference.cpp@148: TRT ERROR: 3: Cannot find binding of given name: [component_container_mt-1] 2023-11-30 19:34:53.109 ERROR /home/jenkins/agent/workspace/ly_debian-compatible-release-2.1/isaac_ros-dev/ros_ws/src/isaac_ros_dnn_inference/isaac_ros_tensor_rt/gxf/tensor_rt/tensor_rt_inference.cpp@341: Failed to get binding index for input in model /tmp/s_best.plan [component_container_mt-1] 2023-11-30 19:34:53.111 WARN gxf/std/multi_thread_scheduler.cpp@325: Error while executing entity E254 named 'CVQLNEENCN_inference': GXF_FAILURE [component_container_mt-1] 2023-11-30 19:34:53.111 ERROR gxf/std/entity_executor.cpp@563: Entity [CVQLNEENCN_inference] must be in Started, Tick Pending, Ticking or Idle stage before stopping. Current state is StartPending [component_container_mt-1] 2023-11-30 19:34:54.046 ERROR gxf/std/entity_executor.cpp@210: Entity with eid 254 not found! [component_container_mt-1] [WARN] [1701369294.046364375] [tensor_rt]: [NitrosNode] The heartbeat entity (eid=254) was stopped. The graph may have been terminated. [component_container_mt-1] [INFO] [1701369294.051208264] [tensor_rt]: [NitrosNode] Terminating the running application [component_container_mt-1] [INFO] [1701369294.051220509] [tensor_rt]: [NitrosContext] Interrupting GXF... [component_container_mt-1] 2023-11-30 19:34:54.051 ERROR gxf/core/runtime.cpp@1400: Graph interrupt failed with error: GXF_FAILURE [component_container_mt-1] [ERROR] [1701369294.051256202] [tensor_rt]: [NitrosContext] GxfGraphInterrupt Error: GXF_FAILURE [component_container_mt-1] [INFO] [1701369294.051259368] [tensor_rt]: [NitrosContext] Waiting on GXF... [component_container_mt-1] 2023-11-30 19:34:54.051 ERROR gxf/std/program.cpp@556: wait failed. Deactivating... [component_container_mt-1] |==================================================================================================================================================================| [component_container_mt-1] | Job Statistics Report (regular) | [component_container_mt-1] |==================================================================================================================================================================| [component_container_mt-1] | Name | Count | Time (Median - 90% - Max) [ms] | Load (%) | Exec(ms) | Variation (Median - 90% - Max) [ns] | [component_container_mt-1] |------------------------------------------------------------------------------------------------------------------------------------------------------------------| [component_container_mt-1] |==================================================================================================================================================================| [component_container_mt-1] |==================================================================================================================================================================| [component_container_mt-1] | Entity Statistics Report (regular) | [component_container_mt-1] |==================================================================================================================================================================| [component_container_mt-1] | Entity Name | Entity State | Count | Time (Median - 90% - Max) [ms] | [component_container_mt-1] |------------------------------------------------------------------------------------------------------------------------------------------------------------------| [component_container_mt-1] |==================================================================================================================================================================| [component_container_mt-1] 2023-11-30 19:34:54.052 WARN gxf/std/entity_warden.cpp@433: Component of type nvidia::gxf::MultiThreadScheduler, cid 277 failed to deinitialize with code GXF_FAILURE [component_container_mt-1] 2023-11-30 19:34:54.052 ERROR gxf/core/runtime.cpp@731: Could not deinitialize entity 'CVQLNEENCN_utils' (E275): GXF_FAILURE [component_container_mt-1] 2023-11-30 19:34:54.052 ERROR gxf/std/program.cpp@558: Deactivation failed. [component_container_mt-1] 2023-11-30 19:34:54.052 ERROR gxf/core/runtime.cpp@1408: Graph wait failed with error: GXF_FAILURE [component_container_mt-1] [ERROR] [1701369294.052356975] [tensor_rt]: [NitrosContext] GxfGraphWait Error: GXF_FAILURE [component_container_mt-1] [INFO] [1701369294.052361284] [tensor_rt]: [NitrosNode] Application termination done [component_container_mt-1] [INFO] [1701369294.055715099] [dnn_image_encoder]: [NitrosNode] Terminating the running application [component_container_mt-1] [INFO] [1701369294.055723036] [dnn_image_encoder]: [NitrosContext] Interrupting GXF... [component_container_mt-1] [INFO] [1701369294.056752561] [dnn_image_encoder]: [NitrosContext] Waiting on GXF... [component_container_mt-1] [INFO] [1701369294.056760362] [dnn_image_encoder]: [NitrosContext] Deinitializing... [component_container_mt-1] [INFO] [1701369294.063464597] [dnn_image_encoder]: [NitrosContext] Destroying context [ERROR] [component_container_mt-1]: process has died [pid 317496, exit code -11, cmd '/opt/ros/humble/lib/rclcpp_components/component_container_mt --ros-args --log-level INFO --ros-args -r __node:=tensor_rt_container -r __ns:=/'].

TeamRoboTo avatar Nov 30 '23 18:11 TeamRoboTo

Hi @TeamRoboTo,

I believe the issue is because the input_binding_names and output_binding_names parameter aren't set.

In the quickstart, the command we provide is:

ros2 launch isaac_ros_yolov8 isaac_ros_yolov8_visualize.launch.py model_file_path:=/tmp/yolov8s.onnx engine_file_path:=/tmp/yolov8s.plan input_binding_names:=['images'] output_binding_names:=['output0'] network_image_width:=640 network_image_height:=640 force_engine_update:=False image_mean:=[0.0,0.0,0.0] image_stddev:=[1.0,1.0,1.0] input_image_width:=640 input_image_height:=640 confidence_threshold:=0.25 nms_threshold:=0.45

The input_binding_names is set to images and the output_binding_names is set to output0. This matches the screenshot you provided of your custom model, so can you try setting those parameters accordingly?

kajananchinniahNV avatar Nov 30 '23 19:11 kajananchinniahNV

Yep, thanks you. Now it works :)

TeamRoboTo avatar Dec 02 '23 14:12 TeamRoboTo

Sorry,I have the same issue.I changed the code cloned from git(num_classes), and then I found that running ros2 launch according to quickstart didn't seem to compile my code, so my changes won't take effect,I want to know the correct operation,any suggestions?

magic-Alex avatar Dec 18 '23 06:12 magic-Alex

Hi @magic-Alex

In our latest release because of our usage of debians now, it's harder to update the source code directly to apply this fix now.

My recommendation for now would be to uninstall the debians for this package and instead compile the code from source (via colcon build) by cloning the repository in your local workspace.

If you run into issues doing this, please let us know!

kajananchinniahNV avatar Dec 21 '23 04:12 kajananchinniahNV

@kajananchinniahNV Thanks,it works!

magic-Alex avatar Dec 21 '23 13:12 magic-Alex

I believe I'm having the same issue. I'm trying to run a model with the following input/output dimensions: image I've already updated my yolov8_decoder_node.cpp to match the amount of classes I have (8), as well as isaac_ros_yolov8_visualizer.py to the correct class names. The example ONNX model is able to run when I follow the Quickstart Guide, but when I try to run my own model, I get the same error in which the process dies. I'm using the following command:

ros2 launch isaac_ros_yolov8 isaac_ros_yolov8_visualize.launch.py model_file_path:=/workspaces/isaac_ros-dev/models/12_1_23-87_8.onnx engine_file_path:=/workspaces/isaac_ros-dev/models/12_1_23-87_8.plan input_binding_names:=['images'] output_binding_names:=['output0'] network_image_width:=640 network_image_height:=640 force_engine_update:=False image_mean:=[0.0,0.0,0.0] image_stddev:=[1.0,1.0,1.0] input_image_width:=640 input_image_height:=640 confidence_threshold:=0.25 nms_threshold:=0.45

Is there anything I'm missing here?

eddieliao avatar Jan 06 '24 04:01 eddieliao

Hi @eddieliao ,

Could you please provide the full error trace? There can be many different reasons that the process dies.

jaiveersinghNV avatar Jan 08 '24 23:01 jaiveersinghNV

I've managed to get it to work with an image, but if I input a video it will crash when the video ends.

Could you please provide the full error trace? There can be many different reasons that the process dies.

Here's the full output, although it doesn't seem too useful?

admin@eddie-Slim-7-ProX-14ARH7:/workspaces/isaac_ros-dev$ ros2 launch isaac_ros_yolov8 isaac_ros_yolov8_visualize.launch.py model_file_path:=/workspaces/isaac_ros-dev/models/12_1_23-87_8.onnx engine_file_path:=/workspaces/isaac_ros-dev/models/12_1_23-87_8.plan input_binding_names:=['images'] output_binding_names:=['output0'] network_image_width:=640 network_image_height:=640 force_engine_update:=False image_mean:=[0.0,0.0,0.0] image_stddev:=[1.0,1.0,1.0] input_image_width:=640 input_image_height:=640 confidence_threshold:=0.25 nms_threshold:=0.45
[INFO] [launch]: All log files can be found below /home/admin/.ros/log/2024-01-12-15-54-05-242222-eddie-Slim-7-ProX-14ARH7-21621
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container_mt-1]: process started with pid [21632]
[INFO] [isaac_ros_yolov8_visualizer.py-2]: process started with pid [21634]
[INFO] [rqt_image_view-3]: process started with pid [21636]
[component_container_mt-1] [INFO] [1705103645.715825702] [tensor_rt_container]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_dnn_image_encoder/lib/libdnn_image_encoder_node.so
[component_container_mt-1] [INFO] [1705103645.750832265] [tensor_rt_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::DnnImageEncoderNode>
[component_container_mt-1] [INFO] [1705103645.750880569] [tensor_rt_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::DnnImageEncoderNode>
[component_container_mt-1] [INFO] [1705103645.754788087] [NitrosContext]: [NitrosContext] Creating a new shared context
[component_container_mt-1] [INFO] [1705103645.754971920] [dnn_image_encoder]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1705103645.755961694] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/std/libgxf_std.so
[component_container_mt-1] [INFO] [1705103645.767465581] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_gxf_helpers.so
[component_container_mt-1] [INFO] [1705103645.774823463] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_sight.so
[component_container_mt-1] [INFO] [1705103645.785412771] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/libgxf_atlas.so
[component_container_mt-1] [INFO] [1705103645.793653142] [NitrosContext]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_nitros/share/isaac_ros_nitros/config/type_adapter_nitros_context_graph.yaml'
[component_container_mt-1] [INFO] [1705103645.795100962] [NitrosContext]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1705103645.802402336] [NitrosContext]: [NitrosContext] Running application...
[component_container_mt-1] 2024-01-12 15:54:05.802 WARN  gxf/std/program.cpp@514: No system specified. Nothing to do
[component_container_mt-1] [INFO] [1705103645.803663739] [dnn_image_encoder]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1705103645.803689839] [dnn_image_encoder]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1705103645.806399723] [dnn_image_encoder]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1705103645.806423008] [dnn_image_encoder]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1705103645.807594502] [dnn_image_encoder]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1705103645.807615324] [dnn_image_encoder]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1705103645.807975499] [dnn_image_encoder]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1705103645.808308812] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/lib/libgxf_message_compositor.so
[component_container_mt-1] [INFO] [1705103645.810069153] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/lib/cuda/libgxf_cuda.so
[component_container_mt-1] [INFO] [1705103645.811678932] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/lib/serialization/libgxf_serialization.so
[component_container_mt-1] [INFO] [1705103645.821224854] [dnn_image_encoder]: [NitrosContext] Loading extension: gxf/lib/image_proc/libgxf_tensorops.so
[component_container_mt-1] [INFO] [1705103645.830523104] [NitrosContext]: [NitrosContext] Loading extension: gxf/lib/multimedia/libgxf_multimedia.so
[component_container_mt-1] [INFO] [1705103645.832671653] [dnn_image_encoder]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1705103645.839784417] [dnn_image_encoder]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1705103646.052297749] [dnn_image_encoder]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1705103646.057241107] [dnn_image_encoder]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1705103646.083560457] [dnn_image_encoder]: [NitrosNode] Starting negotiation...
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/dnn_image_encoder' in container '/tensor_rt_container'
[component_container_mt-1] [INFO] [1705103646.087228880] [tensor_rt_container]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_tensor_rt/lib/libtensor_rt_node.so
[component_container_mt-1] [INFO] [1705103646.142498865] [tensor_rt_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::TensorRTNode>
[component_container_mt-1] [INFO] [1705103646.142570596] [tensor_rt_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::dnn_inference::TensorRTNode>
[component_container_mt-1] [INFO] [1705103646.146771299] [tensor_rt]: [NitrosNode] Initializing NitrosNode
[component_container_mt-1] [INFO] [1705103646.147795758] [tensor_rt]: [NitrosNode] Starting NitrosNode
[component_container_mt-1] [INFO] [1705103646.147810982] [tensor_rt]: [NitrosNode] Loading built-in preset extension specs
[component_container_mt-1] [INFO] [1705103646.149321520] [tensor_rt]: [NitrosNode] Loading built-in extension specs
[component_container_mt-1] [INFO] [1705103646.149339387] [tensor_rt]: [NitrosNode] Loading preset extension specs
[component_container_mt-1] [INFO] [1705103646.149648684] [tensor_rt]: [NitrosNode] Loading extension specs
[component_container_mt-1] [INFO] [1705103646.149666391] [tensor_rt]: [NitrosNode] Loading generator rules
[component_container_mt-1] [INFO] [1705103646.154810539] [tensor_rt]: [NitrosNode] Loading extensions
[component_container_mt-1] [INFO] [1705103646.155955924] [tensor_rt]: [NitrosContext] Loading extension: gxf/tensor_rt/libgxf_tensor_rt.so
[component_container_mt-1] [INFO] [1705103646.158797863] [tensor_rt]: [NitrosNode] Loading graph to the optimizer
[component_container_mt-1] [INFO] [1705103646.160072357] [tensor_rt]: [NitrosNode] Running optimization
[component_container_mt-1] [INFO] [1705103646.169617437] [tensor_rt]: [NitrosNode] Obtaining graph IO group info from the optimizer
[component_container_mt-1] [INFO] [1705103646.170398140] [tensor_rt]: [NitrosNode] Creating negotiated publishers/subscribers
[component_container_mt-1] [INFO] [1705103646.170428117] [tensor_rt]: [NitrosPublisherSubscriberGroup] Pinning the component "inference/rx" (type="nvidia::gxf::DoubleBufferReceiver") to use its compatible format only: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1705103646.170461167] [tensor_rt]: [NitrosPublisherSubscriberGroup] Pinning the component "sink/sink" (type="nvidia::isaac_ros::MessageRelay") to use its compatible format only: "nitros_tensor_list_nhwc_rgb_f32"
[component_container_mt-1] [INFO] [1705103646.172971936] [tensor_rt]: [NitrosNode] Starting negotiation...
[component_container_mt-1] [INFO] [1705103646.173531388] [dnn_image_encoder]: Negotiating
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/tensor_rt' in container '/tensor_rt_container'
[component_container_mt-1] [INFO] [1705103646.175685458] [tensor_rt_container]: Load Library: /workspaces/isaac_ros-dev/install/isaac_ros_yolov8/lib/libyolov8_decoder_node.so
[component_container_mt-1] [INFO] [1705103646.183410443] [tensor_rt_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::yolov8::YoloV8DecoderNode>
[component_container_mt-1] [INFO] [1705103646.183476084] [tensor_rt_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::yolov8::YoloV8DecoderNode>
[component_container_mt-1] [INFO] [1705103646.190048002] [yolov8_decoder_node.ManagedNitrosSubscriber]: Starting Managed Nitros Subscriber
[component_container_mt-1] [INFO] [1705103646.190094553] [tensor_rt]: Negotiating
[component_container_mt-1] [INFO] [1705103646.190141345] [tensor_rt]: Could not negotiate
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/yolov8_decoder_node' in container '/tensor_rt_container'
[component_container_mt-1] [INFO] [1705103647.083791693] [dnn_image_encoder]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1705103647.083879789] [dnn_image_encoder]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1705103647.083900861] [dnn_image_encoder]: [NitrosPublisher] Use the negotiated data format: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1705103647.083916796] [dnn_image_encoder]: [NitrosSubscriber] Negotiation ended with no results
[component_container_mt-1] [INFO] [1705103647.083926721] [dnn_image_encoder]: [NitrosSubscriber] Use the compatible subscriber: topic_name="/image", data_format="nitros_image_bgr8"
[component_container_mt-1] [INFO] [1705103647.083988406] [dnn_image_encoder]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1705103647.150636731] [dnn_image_encoder]: [NitrosNode] Wrote the final top level YAML graph to "/workspaces/isaac_ros-dev/install/isaac_ros_dnn_image_encoder/share/isaac_ros_dnn_image_encoder/VCAEGTJSZK.yaml"
[component_container_mt-1] [INFO] [1705103647.150709203] [dnn_image_encoder]: [NitrosNode] Calling user's pre-load-graph callback
[component_container_mt-1] [INFO] [1705103647.150723936] [dnn_image_encoder]: In DNN Image Encoder Node preLoadGraphCallback().
[component_container_mt-1] [INFO] [1705103647.150757497] [dnn_image_encoder]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1705103647.150765119] [dnn_image_encoder]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_dnn_image_encoder/share/isaac_ros_dnn_image_encoder/VCAEGTJSZK.yaml'
[component_container_mt-1] [INFO] [1705103647.156145866] [dnn_image_encoder]: [NitrosNode] Linking Nitros pub/sub to the loaded application
[component_container_mt-1] [INFO] [1705103647.156314617] [dnn_image_encoder]: [NitrosNode] Calling user's post-load-graph callback
[component_container_mt-1] [INFO] [1705103647.156323510] [dnn_image_encoder]: In DNN Image Encoder Node postLoadGraphCallback().
[component_container_mt-1] [INFO] [1705103647.156402913] [dnn_image_encoder]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1705103647.173021740] [tensor_rt]: [NitrosNode] Starting post negotiation setup
[component_container_mt-1] [INFO] [1705103647.173133532] [tensor_rt]: [NitrosNode] Getting data format negotiation results
[component_container_mt-1] [INFO] [1705103647.173155686] [tensor_rt]: [NitrosPublisher] Negotiation ended with no results
[component_container_mt-1] [INFO] [1705103647.173225053] [tensor_rt]: [NitrosPublisher] Use only the compatible publisher: topic_name="/tensor_sub", data_format="nitros_tensor_list_nhwc_rgb_f32"
[component_container_mt-1] [INFO] [1705103647.173242230] [tensor_rt]: [NitrosSubscriber] Use the negotiated data format: "nitros_tensor_list_nchw_rgb_f32"
[component_container_mt-1] [INFO] [1705103647.173266587] [tensor_rt]: [NitrosNode] Exporting the final graph based on the negotiation results
[component_container_mt-1] [INFO] [1705103647.176585816] [tensor_rt]: [NitrosNode] Wrote the final top level YAML graph to "/workspaces/isaac_ros-dev/install/isaac_ros_tensor_rt/share/isaac_ros_tensor_rt/XJCBXQQGNJ.yaml"
[component_container_mt-1] [INFO] [1705103647.176629843] [tensor_rt]: [NitrosNode] Calling user's pre-load-graph callback
[component_container_mt-1] [INFO] [1705103647.176638046] [tensor_rt]: [NitrosNode] Loading application
[component_container_mt-1] [INFO] [1705103647.244900521] [dnn_image_encoder]: [NitrosContext] Running application...
[component_container_mt-1] [INFO] [1705103647.245189436] [dnn_image_encoder]: [NitrosNode] Starting a heartbeat timer (eid=25)
[component_container_mt-1] [INFO] [1705103647.245216378] [tensor_rt]: [NitrosContext] Loading application: '/workspaces/isaac_ros-dev/install/isaac_ros_tensor_rt/share/isaac_ros_tensor_rt/XJCBXQQGNJ.yaml'
[component_container_mt-1] 2024-01-12 15:54:07.249 WARN  gxf/std/yaml_file_loader.cpp@1058: Using unregistered parameter 'dummy_rx' in component ''.
[component_container_mt-1] 2024-01-12 15:54:07.250 WARN  gxf/std/yaml_file_loader.cpp@1058: Using unregistered parameter 'dev_id' in component 'stream'.
[component_container_mt-1] [INFO] [1705103647.250522549] [tensor_rt]: [NitrosNode] Linking Nitros pub/sub to the loaded application
[component_container_mt-1] [INFO] [1705103647.250671239] [tensor_rt]: [NitrosNode] Calling user's post-load-graph callback
[component_container_mt-1] [INFO] [1705103647.250687013] [tensor_rt]: In TensorRTNode postLoadGraphCallback().
[component_container_mt-1] [INFO] [1705103647.494105456] [tensor_rt]: Read tensor shape information from TRT Model Engine: /workspaces/isaac_ros-dev/models/12_1_23-87_8.plan
[component_container_mt-1] [INFO] [1705103647.494248387] [tensor_rt]: Tensors 403200 bytes, num outputs 40 x tensors per output 1 = 40 blocks
[component_container_mt-1] [INFO] [1705103647.494363895] [tensor_rt]: [NitrosContext] Initializing application...
[component_container_mt-1] [INFO] [1705103647.501373574] [tensor_rt]: [NitrosContext] Running application...
[component_container_mt-1] [INFO] [1705103647.501617660] [tensor_rt]: [NitrosNode] Starting a heartbeat timer (eid=112)
[ERROR] [component_container_mt-1]: process has died [pid 21632, exit code -11, cmd '/opt/ros/humble/lib/rclcpp_components/component_container_mt --ros-args --log-level INFO --ros-args -r __node:=tensor_rt_container -r __ns:=/'].```

eddieliao avatar Jan 12 '24 23:01 eddieliao