NVIDIA
Internal Error (Could not find any implementation for node model/backbones.0/backbones.0.0/body/conv1/Conv + onnx::Mul_3979 + model/backbones.0/backbones.0.0/body/bn1/Mul + onnx::Add_3981 + model/backbones.0/backbones.0.0/body/bn1/Add + model/backbones.0/backbones.0.0/body/relu/Relu.
Description
Attempting to convert an ONNX model to a TensorRT engine using the trtexec tool fails, with errors indicating that no kernel image is available for the device and that no implementation can be found for the model's convolutional layer and subsequent batch normalization/activation layers. Specific errors include:
-
no kernel image is available for execution on the device
-
Could not find any implementation for node model/backbones.0/backbones.0.0/body/conv1/Conv + ... + Relu.
Environment
TensorRT Version: 10.3.0.30 (arm64 architecture, compatible with CUDA 12.6) NVIDIA GPU: NVIDIA Jetson AGX Orin Developer Kit (Ampere architecture, compute capability 8.7, 64GB RAM) NVIDIA Driver Version: 540.4.0 CUDA Version: 12.6.68 CUDNN Version: Not explicitly shown (not found via dpkg, presumably installed by default with TensorRT 10.3)
Operating System: Ubuntu 22.04.5 LTS (Jammy Jellyfish), Jetson Linux (aarch64 architecture, kernel 5.15.148-tegra) Python Version: 3.10.17 PyTorch Version: 2.5.0 Baremetal: Yes (non-container environment)
Relevant Files
Model link: Local model coffee_machine_policy_simple.onnx (simplified ONNX model based on CNN architecture, containing convolution, batch normalization, ReLU, etc.)
Steps To Reproduce
Commands or scripts:
- Simplify the ONNX model: python -m onnxsim coffee_machine_policy.onnx coffee_machine_policy_simple.onnx
- Convert to TensorRT engine: trtexec --onnx=coffee_machine_policy_simple.onnx --saveEngine=coffee_machine_policy.engine --verbose
Full traceback of errors encountered:
plaintext [09/04/2025-16:30:23] [E] Error[9]: Error Code: 9: Skipping tactic 0xf067e6205da31c2e due to exception initDeviceReservedSpace [09/04/2025-16:30:24] [E] Error[9]: Error Code: 9: Skipping tactic 0x5deb29b7a8e275f7 due to exception initDeviceReservedSpace [09/04/2025-16:30:24] [E] Error[9]: Error Code: 9: Skipping tactic 0xf64396b97c889179 due to exception initDeviceReservedSpace [09/04/2025-16:30:24] [E] Error[9]: Error Code: 9: Skipping tactic 0xbdfdef6b84f7ccc9 due to exception initDeviceReservedSpace [09/04/2025-16:30:24] [E] Error[9]: Error Code: 9: Skipping tactic 0x7bc32c782b800c48 due to exception initDeviceReservedSpace [09/04/2025-16:30:24] [E] Error[10]: IBuilder::buildSerializedNetwork: Error Code 10: Internal Error (Could not find any implementation for node model/backbones.0/backbones.0.0/body/conv1/Conv + onnx::Mul_3979 + model/backbones.0/backbones.0.0/body/bn1/Mul + onnx::Add_3981 + model/backbones.0/backbones.0.0/body/bn1/Add + model/backbones.0/backbones.0.0/body/relu/Relu.) [09/04/2025-16:30:24] [E] Error[1]: [checkMacros.cpp::catchCudaError::205] Error Code 1: Cuda Runtime (no kernel image is available for execution on the device) [09/04/2025-16:30:24] [E] Engine could not be created from network [09/04/2025-16:30:24] [E] Building engine failed [09/04/2025-16:30:24] [E] Failed to create engine from model or file. [09/04/2025-16:30:24] [E] Engine set up failed
Have you tried the latest release?: Yes, using the latest TensorRT 10.3 and PyTorch 2.5.0 Can this model run on other frameworks?: Yes, the ONNX model runs successfully on Jetson via ONNX Runtime (verified with polygraphy run coffee_machine_policy_simple.onnx --onnxrt)
Could you provide the ONNX model file? Is this the name of a single node?
model/backbones.0/backbones.0.0/body/conv1/Conv + onnx::Mul_3979 + model/backbones.0/backbones.0.0/body/bn1/Mul + onnx::Add_3981 + model/backbones.0/backbones.0.0/body/bn1/Add + model/backbones.0/backbones.0.0/body/relu/Relu
If this is the particular naming, perhaps the ONNX parser thinks it is a custom op and is not recognizing the individual operations. However, it should be able to parse individual Conv, Mul, Add, ReLU nodes and then the operations should be fused in the final engine.
model/backbones.0/backbones.0.0/body/conv1/Conv The ONNX model is a bit large, so I'll first provide this partial screenshot. Yes, this is a single node.