onnx-tensorrt
onnx-tensorrt copied to clipboard
onnx
Problem converting ONNX model to TensorRT 7.2.1(dien.onnx)
when i tried convert onnx to tensorrt (trtexec --onnx=dien.onnx) , it occurs error:
Input filename: dien.onnx ONNX IR version: 0.0.7 Opset version: 13 Producer name: tf2onnx Producer version: 1.9.0 Domain: Model version: 0 Doc string:
[04/20/2021-11:25:41] [W] [TRT] onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. terminate called after throwing an instance of 'std::out_of_range' what(): Attribute not found: axes Aborted (core dumped)
when i tried "trtexec --onnx=dien.onnx --verbose", it occurs : [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:119: Searching for input: rnn_2/gru2/while/sub/x:0 [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:119: Searching for input: Attention_layer_1/ones_like__51:0 [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:125: Attention_layer_1/ones_like [Expand] inputs: [rnn_2/gru2/while/sub/x:0 -> ()], [Attention_layer_1/ones_like__51:0 -> (2)], [04/20/2021-11:28:22] [V] [TRT] ImporterContext.hpp:141: Registering layer: Attention_layer_1/ones_like for ONNX node: Attention_layer_1/ones_like [04/20/2021-11:28:22] [V] [TRT] ImporterContext.hpp:116: Registering tensor: Attention_layer_1/ones_like:0 for ONNX tensor: Attention_layer_1/ones_like:0 [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:179: Attention_layer_1/ones_like [Expand] outputs: [Attention_layer_1/ones_like:0 -> (-1, -1)], [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:103: Parsing node: Attention_layer_1/f11_1/MatMul [MatMul] [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:119: Searching for input: concat:0 [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:119: Searching for input: f11_1/kernel/read/14__cf__14:0 [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:125: Attention_layer_1/f11_1/MatMul [MatMul] inputs: [concat:0 -> (-1, 36)], [f11_1/kernel/read/14__cf__14:0 -> (36, 36)], [04/20/2021-11:28:22] [V] [TRT] builtin_op_importers.cpp:2053: GEMM: using FC layer instead of MM because all criteria were met. [04/20/2021-11:28:22] [V] [TRT] onnx2trt_utils.cpp:1793: Original shape: (, 36), unsqueezing to: (, _, , ) [04/20/2021-11:28:22] [V] [TRT] ImporterContext.hpp:141: Registering layer: Attention_layer_1/f11_1/MatMul for ONNX node: Attention_layer_1/f11_1/MatMul [04/20/2021-11:28:22] [V] [TRT] onnx2trt_utils.cpp:1641: Original shape: (, 36, 1, 1), squeezing to: (, _) [04/20/2021-11:28:22] [V] [TRT] ImporterContext.hpp:116: Registering tensor: Attention_layer_1/f11_1/MatMul:0 for ONNX tensor: Attention_layer_1/f11_1/MatMul:0 [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:179: Attention_layer_1/f11_1/MatMul [MatMul] outputs: [Attention_layer_1/f11_1/MatMul:0 -> (-1, -1)], [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:103: Parsing node: Attention_layer_1/f11_1/BiasAdd [Add] [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:119: Searching for input: Attention_layer_1/f11_1/MatMul:0 [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:119: Searching for input: f11_1/bias/read/_13__cf__13:0 [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:125: Attention_layer_1/f11_1/BiasAdd [Add] inputs: [Attention_layer_1/f11_1/MatMul:0 -> (-1, -1)], [f11_1/bias/read/_13__cf__13:0 -> (36)], [04/20/2021-11:28:22] [V] [TRT] ImporterContext.hpp:141: Registering layer: Attention_layer_1/f11_1/BiasAdd for ONNX node: Attention_layer_1/f11_1/BiasAdd [04/20/2021-11:28:22] [V] [TRT] ImporterContext.hpp:116: Registering tensor: Attention_layer_1/f11_1/BiasAdd:0 for ONNX tensor: Attention_layer_1/f11_1/BiasAdd:0 [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:179: Attention_layer_1/f11_1/BiasAdd [Add] outputs: [Attention_layer_1/f11_1/BiasAdd:0 -> (-1, 36)], [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:103: Parsing node: rnn_1/gru1/GRUCellZeroState/zeros [Expand] [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:119: Searching for input: Attention_layer_1/Minimum/x:0 [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:119: Searching for input: rnn_1/gru1/GRUCellZeroState/zeros__36:0 [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:125: rnn_1/gru1/GRUCellZeroState/zeros [Expand] inputs: [Attention_layer_1/Minimum/x:0 -> ()], [rnn_1/gru1/GRUCellZeroState/zeros__36:0 -> (2)], [04/20/2021-11:28:22] [V] [TRT] ImporterContext.hpp:141: Registering layer: rnn_1/gru1/GRUCellZeroState/zeros for ONNX node: rnn_1/gru1/GRUCellZeroState/zeros [04/20/2021-11:28:22] [V] [TRT] ImporterContext.hpp:116: Registering tensor: rnn_1/gru1/GRUCellZeroState/zeros:0 for ONNX tensor: rnn_1/gru1/GRUCellZeroState/zeros:0 [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:179: rnn_1/gru1/GRUCellZeroState/zeros [Expand] outputs: [rnn_1/gru1/GRUCellZeroState/zeros:0 -> (-1, -1)], [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:103: Parsing node: Unsqueeze__13 [Unsqueeze] [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:119: Searching for input: rnn_1/gru1/GRUCellZeroState/zeros:0 [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:119: Searching for input: const_axes__45 [04/20/2021-11:28:22] [V] [TRT] ModelImporter.cpp:125: Unsqueeze__13 [Unsqueeze] inputs: [rnn_1/gru1/GRUCellZeroState/zeros:0 -> (-1, -1)], [const_axes__45 -> (1)], terminate called after throwing an instance of 'std::out_of_range' what(): Attribute not found: axes
It seems tensorrt convert int64 to int32, which incurs index . How can I solve the problem? Thanks for help.
It looks like you are using the opset 13 version of Unsqueeze which is currently unsupported. Can you try exporting your model to a lower opset (i.e. opset 11)?
Thanks for reply.When i change to op11,it occurs:
[05/27/2021-04:47:17] [V] [TRT] ModelImporter.cpp:119: Searching for input: add_8:0 [05/27/2021-04:47:17] [V] [TRT] ModelImporter.cpp:119: Searching for input: f3/kernel/read/20__cf__20:0 [05/27/2021-04:47:17] [V] [TRT] ModelImporter.cpp:125: f3/MatMul [MatMul] inputs: [add_8:0 -> (-1, 80)], [f3/kernel/read/20__cf__20:0 -> (80, 2)], [05/27/2021-04:47:17] [V] [TRT] builtin_op_importers.cpp:2053: GEMM: using FC layer instead of MM because all criteria were met. [05/27/2021-04:47:17] [V] [TRT] onnx2trt_utils.cpp:1793: Original shape: (, 80), unsqueezing to: (, _, , ) [05/27/2021-04:47:17] [V] [TRT] ImporterContext.hpp:141: Registering layer: f3/MatMul for ONNX node: f3/MatMul [05/27/2021-04:47:17] [V] [TRT] onnx2trt_utils.cpp:1641: Original shape: (, 2, 1, 1), squeezing to: (, _) [05/27/2021-04:47:17] [V] [TRT] ImporterContext.hpp:116: Registering tensor: f3/MatMul:0 for ONNX tensor: f3/MatMul:0 [05/27/2021-04:47:17] [V] [TRT] ModelImporter.cpp:179: f3/MatMul [MatMul] outputs: [f3/MatMul:0 -> (-1, -1)], [05/27/2021-04:47:17] [V] [TRT] ModelImporter.cpp:103: Parsing node: f3/BiasAdd [Add] [05/27/2021-04:47:17] [V] [TRT] ModelImporter.cpp:119: Searching for input: f3/MatMul:0 [05/27/2021-04:47:17] [V] [TRT] ModelImporter.cpp:119: Searching for input: f3/bias/read/_19__cf__19:0 [05/27/2021-04:47:17] [V] [TRT] ModelImporter.cpp:125: f3/BiasAdd [Add] inputs: [f3/MatMul:0 -> (-1, -1)], [f3/bias/read/_19__cf__19:0 -> (2)], [05/27/2021-04:47:17] [V] [TRT] ImporterContext.hpp:141: Registering layer: f3/BiasAdd for ONNX node: f3/BiasAdd [05/27/2021-04:47:17] [V] [TRT] ImporterContext.hpp:116: Registering tensor: f3/BiasAdd:0 for ONNX tensor: f3/BiasAdd:0 [05/27/2021-04:47:17] [V] [TRT] ModelImporter.cpp:179: f3/BiasAdd [Add] outputs: [f3/BiasAdd:0 -> (-1, 2)], [05/27/2021-04:47:17] [V] [TRT] ModelImporter.cpp:103: Parsing node: Softmax_2 [Softmax] [05/27/2021-04:47:17] [V] [TRT] ModelImporter.cpp:119: Searching for input: f3/BiasAdd:0 [05/27/2021-04:47:17] [V] [TRT] ModelImporter.cpp:125: Softmax_2 [Softmax] inputs: [f3/BiasAdd:0 -> (-1, 2)], [05/27/2021-04:47:17] [V] [TRT] ImporterContext.hpp:141: Registering layer: Softmax_2 for ONNX node: Softmax_2 [05/27/2021-04:47:17] [V] [TRT] ImporterContext.hpp:116: Registering tensor: Softmax_2:0 for ONNX tensor: Softmax_2:0 [05/27/2021-04:47:17] [V] [TRT] ModelImporter.cpp:179: Softmax_2 [Softmax] outputs: [Softmax_2:0 -> (-1, -1)], [05/27/2021-04:47:17] [V] [TRT] ModelImporter.cpp:103: Parsing node: add_9 [Add] [05/27/2021-04:47:17] [V] [TRT] ModelImporter.cpp:119: Searching for input: Softmax_2:0 [05/27/2021-04:47:17] [V] [TRT] ModelImporter.cpp:119: Searching for input: add_9/y:0 [05/27/2021-04:47:17] [V] [TRT] ModelImporter.cpp:125: add_9 [Add] inputs: [Softmax_2:0 -> (-1, -1)], [add_9/y:0 -> ()], [05/27/2021-04:47:17] [V] [TRT] ImporterContext.hpp:141: Registering layer: add_9 for ONNX node: add_9 [05/27/2021-04:47:17] [V] [TRT] ImporterContext.hpp:116: Registering tensor: add_9:0_1 for ONNX tensor: add_9:0 [05/27/2021-04:47:17] [V] [TRT] ModelImporter.cpp:179: add_9 [Add] outputs: [add_9:0 -> (-1, -1)], [05/27/2021-04:47:17] [V] [TRT] ModelImporter.cpp:507: Marking add_9:0_1 as output: add_9:0 [05/27/2021-04:47:17] [E] [TRT] ModelImporter.cpp:463: Found rnn_2/gru2/while/Identity:0 as a shape tensor output from a layer that does not support it! [05/27/2021-04:47:17] [W] [TRT] Output type must be INT32 for shape outputs [05/27/2021-04:47:17] [W] [TRT] Output type must be INT32 for shape outputs [05/27/2021-04:47:17] [W] [TRT] Output type must be INT32 for shape outputs [05/27/2021-04:47:17] [W] [TRT] Output type must be INT32 for shape outputs [05/27/2021-04:47:17] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output. [05/27/2021-04:47:17] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output. [05/27/2021-04:47:17] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output. [05/27/2021-04:47:17] [W] [TRT] Output type must be INT32 for shape outputs [05/27/2021-04:47:17] [W] [TRT] Output type must be INT32 for shape outputs ----- Parsing of ONNX model dien_op11.onnx is Done ---- [05/27/2021-04:47:17] [E] [TRT] (Unnamed Layer* 396) [Constant]: cannot use precision Int32 with weights of type Float [05/27/2021-04:47:17] [E] [TRT] Layer (Unnamed Layer* 396) [Constant] failed validation [05/27/2021-04:47:17] [E] [TRT] Network validation failed. [05/27/2021-04:47:17] [E] Engine creation failed [05/27/2021-04:47:17] [E] Engine set up failed &&&& FAILED TensorRT.trtexec # trtexec --onnx=dien_op11.onnx --verbose
It seems " Found rnn_2/gru2/while/Identity:0 as a shape tensor output from a layer that does not support it!", shall I remove identity?
I tried opset with 11，occurs error： [Recurrence]'s output can not be used as shape tensor
&&&& RUNNING TensorRT.trtexec # trtexec --onnx=onnx/dien_op11.onnx --minShapes=Inputs/uid_batch_ph:0:1,Inputs/mid_batch_ph:0:1,Inputs/cat_batch_ph:0:1,Inputs/mid_his_batch_ph:0:19,Inputs/cat_his_batch_ph:0:19,Inputs/mask:0:19,Inputs/seq_len_ph:0:1 --optShapes=Inputs/uid_batch_ph:0:1,Inputs/mid_batch_ph:0:1,Inputs/cat_batch_ph:0:1,Inputs/mid_his_batch_ph:0:19,Inputs/cat_his_batch_ph:0:19,Inputs/mask:0:19,Inputs/seq_len_ph:0:1 --maxShapes=Inputs/uid_batch_ph:0:1,Inputs/mid_batch_ph:0:1,Inputs/cat_batch_ph:0:1,Inputs/mid_his_batch_ph:0:19,Inputs/cat_his_batch_ph:0:19,Inputs/mask:0:19,Inputs/seq_len_ph:0:1 --saveEngine=engine/dien_b1_fp32_static.engine [06/16/2021-08:17:03] [I] === Model Options === [06/16/2021-08:17:03] [I] Format: ONNX [06/16/2021-08:17:03] [I] Model: onnx/dien_op11.onnx [06/16/2021-08:17:03] [I] Output: [06/16/2021-08:17:03] [I] === Build Options === [06/16/2021-08:17:03] [I] Max batch: explicit [06/16/2021-08:17:03] [I] Workspace: 16 MiB [06/16/2021-08:17:03] [I] minTiming: 1 [06/16/2021-08:17:03] [I] avgTiming: 8 [06/16/2021-08:17:03] [I] Precision: FP32 [06/16/2021-08:17:03] [I] Calibration: [06/16/2021-08:17:03] [I] Refit: Disabled [06/16/2021-08:17:03] [I] Safe mode: Disabled [06/16/2021-08:17:03] [I] Save engine: engine/dien_b1_fp32_static.engine [06/16/2021-08:17:03] [I] Load engine: [06/16/2021-08:17:03] [I] Builder Cache: Enabled [06/16/2021-08:17:03] [I] NVTX verbosity: 0 [06/16/2021-08:17:03] [I] Tactic sources: Using default tactic sources [06/16/2021-08:17:03] [I] Input(s)s format: fp32:CHW [06/16/2021-08:17:03] [I] Output(s)s format: fp32:CHW [06/16/2021-08:17:03] [I] Input build shape: Inputs/mask:0=19+19+19 [06/16/2021-08:17:03] [I] Input build shape: Inputs/cat_his_batch_ph:0=19+19+19 [06/16/2021-08:17:03] [I] Input build shape: Inputs/mid_his_batch_ph:0=19+19+19 [06/16/2021-08:17:03] [I] Input build shape: Inputs/cat_batch_ph:0=1+1+1 [06/16/2021-08:17:03] [I] Input build shape: Inputs/mid_batch_ph:0=1+1+1 [06/16/2021-08:17:03] [I] Input build shape: Inputs/seq_len_ph:0=1+1+1 [06/16/2021-08:17:03] [I] Input build shape: Inputs/uid_batch_ph:0=1+1+1 [06/16/2021-08:17:03] [I] Input calibration shapes: model [06/16/2021-08:17:03] [I] === System Options === [06/16/2021-08:17:03] [I] Device: 0 [06/16/2021-08:17:03] [I] DLACore: [06/16/2021-08:17:03] [I] Plugins: [06/16/2021-08:17:03] [I] === Inference Options === [06/16/2021-08:17:03] [I] Batch: Explicit [06/16/2021-08:17:03] [I] Input inference shape: Inputs/uid_batch_ph:0=1 [06/16/2021-08:17:03] [I] Input inference shape: Inputs/seq_len_ph:0=1 [06/16/2021-08:17:03] [I] Input inference shape: Inputs/mid_batch_ph:0=1 [06/16/2021-08:17:03] [I] Input inference shape: Inputs/cat_batch_ph:0=1 [06/16/2021-08:17:03] [I] Input inference shape: Inputs/mid_his_batch_ph:0=19 [06/16/2021-08:17:03] [I] Input inference shape: Inputs/cat_his_batch_ph:0=19 [06/16/2021-08:17:03] [I] Input inference shape: Inputs/mask:0=19 [06/16/2021-08:17:03] [I] Iterations: 10 [06/16/2021-08:17:03] [I] Duration: 3s (+ 200ms warm up) [06/16/2021-08:17:03] [I] Sleep time: 0ms [06/16/2021-08:17:03] [I] Streams: 1 [06/16/2021-08:17:03] [I] ExposeDMA: Disabled [06/16/2021-08:17:03] [I] Data transfers: Enabled [06/16/2021-08:17:03] [I] Spin-wait: Disabled [06/16/2021-08:17:03] [I] Multithreading: Disabled [06/16/2021-08:17:03] [I] CUDA Graph: Disabled [06/16/2021-08:17:03] [I] Separate profiling: Disabled [06/16/2021-08:17:03] [I] Skip inference: Disabled [06/16/2021-08:17:03] [I] Inputs: [06/16/2021-08:17:03] [I] === Reporting Options === [06/16/2021-08:17:03] [I] Verbose: Disabled [06/16/2021-08:17:03] [I] Averages: 10 inferences [06/16/2021-08:17:03] [I] Percentile: 99 [06/16/2021-08:17:03] [I] Dump refittable layers:Disabled [06/16/2021-08:17:03] [I] Dump output: Disabled [06/16/2021-08:17:03] [I] Profile: Disabled [06/16/2021-08:17:03] [I] Export timing to JSON file: [06/16/2021-08:17:03] [I] Export output to JSON file: [06/16/2021-08:17:03] [I] Export profile to JSON file: [06/16/2021-08:17:03] [I] [06/16/2021-08:17:04] [I] === Device Information === [06/16/2021-08:17:04] [I] Selected Device: Tesla T4 [06/16/2021-08:17:04] [I] Compute Capability: 7.5 [06/16/2021-08:17:04] [I] SMs: 40 [06/16/2021-08:17:04] [I] Compute Clock Rate: 1.59 GHz [06/16/2021-08:17:04] [I] Device Global Memory: 15109 MiB [06/16/2021-08:17:04] [I] Shared Memory per SM: 64 KiB [06/16/2021-08:17:04] [I] Memory Bus Width: 256 bits (ECC enabled) [06/16/2021-08:17:04] [I] Memory Clock Rate: 5.001 GHz [06/16/2021-08:17:04] [I] [06/16/2021-08:17:11] [I] [TRT] ---------------------------------------------------------------- [06/16/2021-08:17:11] [I] [TRT] Input filename: onnx/dien_op11.onnx [06/16/2021-08:17:11] [I] [TRT] ONNX IR version: 0.0.6 [06/16/2021-08:17:11] [I] [TRT] Opset version: 11 [06/16/2021-08:17:11] [I] [TRT] Producer name: tf2onnx [06/16/2021-08:17:11] [I] [TRT] Producer version: 1.9.0 [06/16/2021-08:17:11] [I] [TRT] Domain: [06/16/2021-08:17:11] [I] [TRT] Model version: 0 [06/16/2021-08:17:11] [I] [TRT] Doc string: [06/16/2021-08:17:11] [I] [TRT] ---------------------------------------------------------------- [06/16/2021-08:17:11] [W] [TRT] /workspace/TensorRT/t/oss-cicd/oss/parsers/onnx/onnx2trt_utils.cpp:227: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [06/16/2021-08:17:11] [E] [TRT] /workspace/TensorRT/t/oss-cicd/oss/parsers/onnx/ModelImporter.cpp:500: Found unsupported shape tensor producing layer: (Unnamed Layer* 359) [Recurrence] [06/16/2021-08:17:11] [W] [TRT] Output type must be INT32 for shape outputs [06/16/2021-08:17:11] [W] [TRT] Output type must be INT32 for shape outputs [06/16/2021-08:17:11] [W] [TRT] Output type must be INT32 for shape outputs [06/16/2021-08:17:11] [W] [TRT] Output type must be INT32 for shape outputs [06/16/2021-08:17:11] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output. [06/16/2021-08:17:11] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output. [06/16/2021-08:17:11] [W] [TRT] Tensor DataType is determined at build time for tensors not marked as input or output. [06/16/2021-08:17:11] [W] [TRT] Output type must be INT32 for shape outputs [06/16/2021-08:17:11] [W] [TRT] Output type must be INT32 for shape outputs [06/16/2021-08:17:11] [E] [TRT] Layer: (Unnamed Layer* 359) [Recurrence]'s output can not be used as shape tensor. [06/16/2021-08:17:11] [E] [TRT] Network validation failed. [06/16/2021-08:17:11] [E] Engine creation failed [06/16/2021-08:17:11] [E] Engine set up failed &&&& FAILED TensorRT.trtexec # trtexec --onnx=onnx/dien_op11.onnx --minShapes=Inputs/uid_batch_ph:0:1,Inputs/mid_batch_ph:0:1,Inputs/cat_batch_ph:0:1,Inputs/mid_his_batch_ph:0:19,Inputs/cat_his_batch_ph:0:19,Inputs/mask:0:19,Inputs/seq_len_ph:0:1 --optShapes=Inputs/uid_batch_ph:0:1,Inputs/mid_batch_ph:0:1,Inputs/cat_batch_ph:0:1,Inputs/mid_his_batch_ph:0:19,Inputs/cat_his_batch_ph:0:19,Inputs/mask:0:19,Inputs/seq_len_ph:0:1 --maxShapes=Inputs/uid_batch_ph:0:1,Inputs/mid_batch_ph:0:1,Inputs/cat_batch_ph:0:1,Inputs/mid_his_batch_ph:0:19,Inputs/cat_his_batch_ph:0:19,Inputs/mask:0:19,Inputs/seq_len_ph:0:1 --saveEngine=engine/dien_b1_fp32_static.engine