converted ONNX to trt failed( Internal Error (Assertion weights.type() == DataType::kINT32 failed)

[wikipedia2008]

I am getting this output converted ONNX to trt:

&&&& RUNNING TensorRT.trtexec [TensorRT v8201] # /workspace/tensorrt/bin/trtexec --onnx=model.onnx --saveEngine=model.engine [12/03/2021-11:47:34] [I] === Model Options === [12/03/2021-11:47:34] [I] Format: ONNX [12/03/2021-11:47:34] [I] Model: model.onnx [12/03/2021-11:47:34] [I] Output: [12/03/2021-11:47:34] [I] === Build Options === [12/03/2021-11:47:34] [I] Max batch: explicit batch [12/03/2021-11:47:34] [I] Workspace: 16 MiB [12/03/2021-11:47:34] [I] minTiming: 1 [12/03/2021-11:47:34] [I] avgTiming: 8 [12/03/2021-11:47:34] [I] Precision: FP32 [12/03/2021-11:47:34] [I] Calibration: [12/03/2021-11:47:34] [I] Refit: Disabled [12/03/2021-11:47:34] [I] Sparsity: Disabled [12/03/2021-11:47:34] [I] Safe mode: Disabled [12/03/2021-11:47:34] [I] DirectIO mode: Disabled [12/03/2021-11:47:34] [I] Restricted mode: Disabled [12/03/2021-11:47:34] [I] Save engine: model.engine [12/03/2021-11:47:34] [I] Load engine: [12/03/2021-11:47:34] [I] Profiling verbosity: 0 [12/03/2021-11:47:34] [I] Tactic sources: Using default tactic sources [12/03/2021-11:47:34] [I] timingCacheMode: local [12/03/2021-11:47:34] [I] timingCacheFile: [12/03/2021-11:47:34] [I] Input(s)s format: fp32:CHW [12/03/2021-11:47:34] [I] Output(s)s format: fp32:CHW [12/03/2021-11:47:34] [I] Input build shapes: model [12/03/2021-11:47:34] [I] Input calibration shapes: model [12/03/2021-11:47:34] [I] === System Options === [12/03/2021-11:47:34] [I] Device: 0 [12/03/2021-11:47:34] [I] DLACore: [12/03/2021-11:47:34] [I] Plugins: [12/03/2021-11:47:34] [I] === Inference Options === [12/03/2021-11:47:34] [I] Batch: Explicit [12/03/2021-11:47:34] [I] Input inference shapes: model [12/03/2021-11:47:34] [I] Iterations: 10 [12/03/2021-11:47:34] [I] Duration: 3s (+ 200ms warm up) [12/03/2021-11:47:34] [I] Sleep time: 0ms [12/03/2021-11:47:34] [I] Idle time: 0ms [12/03/2021-11:47:34] [I] Streams: 1 [12/03/2021-11:47:34] [I] ExposeDMA: Disabled [12/03/2021-11:47:34] [I] Data transfers: Enabled [12/03/2021-11:47:34] [I] Spin-wait: Disabled [12/03/2021-11:47:34] [I] Multithreading: Disabled [12/03/2021-11:47:34] [I] CUDA Graph: Disabled [12/03/2021-11:47:34] [I] Separate profiling: Disabled [12/03/2021-11:47:34] [I] Time Deserialize: Disabled [12/03/2021-11:47:34] [I] Time Refit: Disabled [12/03/2021-11:47:34] [I] Skip inference: Disabled [12/03/2021-11:47:34] [I] Inputs: [12/03/2021-11:47:34] [I] === Reporting Options === [12/03/2021-11:47:34] [I] Verbose: Disabled [12/03/2021-11:47:34] [I] Averages: 10 inferences [12/03/2021-11:47:34] [I] Percentile: 99 [12/03/2021-11:47:34] [I] Dump refittable layers:Disabled [12/03/2021-11:47:34] [I] Dump output: Disabled [12/03/2021-11:47:34] [I] Profile: Disabled [12/03/2021-11:47:34] [I] Export timing to JSON file: [12/03/2021-11:47:34] [I] Export output to JSON file: [12/03/2021-11:47:34] [I] Export profile to JSON file: [12/03/2021-11:47:34] [I] [12/03/2021-11:47:36] [I] === Device Information === [12/03/2021-11:47:36] [I] Selected Device: Tesla V100-PCIE-16GB [12/03/2021-11:47:36] [I] Compute Capability: 7.0 [12/03/2021-11:47:36] [I] SMs: 80 [12/03/2021-11:47:36] [I] Compute Clock Rate: 1.38 GHz [12/03/2021-11:47:36] [I] Device Global Memory: 16160 MiB [12/03/2021-11:47:36] [I] Shared Memory per SM: 96 KiB [12/03/2021-11:47:36] [I] Memory Bus Width: 4096 bits (ECC enabled) [12/03/2021-11:47:36] [I] Memory Clock Rate: 0.877 GHz [12/03/2021-11:47:36] [I] [12/03/2021-11:47:36] [I] TensorRT version: 8.2.1 [12/03/2021-11:47:36] [I] [TRT] [MemUsageChange] Init CUDA: CPU +250, GPU +0, now: CPU 257, GPU 489 (MiB) [12/03/2021-11:47:36] [I] Start parsing network model [12/03/2021-11:47:36] [I] [TRT] ---------------------------------------------------------------- [12/03/2021-11:47:36] [I] [TRT] Input filename: model.onnx [12/03/2021-11:47:36] [I] [TRT] ONNX IR version: 0.0.7 [12/03/2021-11:47:36] [I] [TRT] Opset version: 12 [12/03/2021-11:47:36] [I] [TRT] Producer name: tf2onnx [12/03/2021-11:47:36] [I] [TRT] Producer version: 1.9.3 [12/03/2021-11:47:36] [I] [TRT] Domain: [12/03/2021-11:47:36] [I] [TRT] Model version: 0 [12/03/2021-11:47:36] [I] [TRT] Doc string: [12/03/2021-11:47:36] [I] [TRT] ---------------------------------------------------------------- [12/03/2021-11:47:36] [W] [TRT] onnx2trt_utils.cpp:362: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [12/03/2021-11:47:36] [W] [TRT] ShapedWeights.cpp:171: Weights StatefulPartitionedCall/x_net/dense/Tensordot/ReadVariableOp:0 has been transposed with permutation of (1, 0)! If you plan on overwriting the weights with the Refitter API, the new weights must be pre-transposed. [12/03/2021-11:47:36] [E] Error[2]: [constantNode.cpp::symbolicExecute::20] Error Code 2: Internal Error (Assertion weights.type() == DataType::kINT32 failed.) [12/03/2021-11:47:36] [E] [TRT] ModelImporter.cpp:720: While parsing node number 40 [Slice -> "StatefulPartitionedCall/x_net/positional_encoding/strided_slice_1:0"]: [12/03/2021-11:47:36] [E] [TRT] ModelImporter.cpp:721: --- Begin node --- [12/03/2021-11:47:36] [E] [TRT] ModelImporter.cpp:722: input: "Func/StatefulPartitionedCall/input/_13:0" input: "begin_masked__45" input: "StatefulPartitionedCall/x_net/positional_encoding/strided_slice_1_Where__59:0" input: "slice_axes__60" input: "StatefulPartitionedCall/x_net/positional_encoding/strided_slice_1/stack_2:0" output: "StatefulPartitionedCall/x_net/positional_encoding/strided_slice_1:0" name: "StatefulPartitionedCall/x_net/positional_encoding/strided_slice_1" op_type: "Slice" domain: ""

[12/03/2021-11:47:36] [E] [TRT] ModelImporter.cpp:723: --- End node --- [12/03/2021-11:47:36] [E] [TRT] ModelImporter.cpp:725: ERROR: ModelImporter.cpp:179 In function parseGraph: [6] Invalid Node - StatefulPartitionedCall/x_net/positional_encoding/strided_slice_1 [constantNode.cpp::symbolicExecute::20] Error Code 2: Internal Error (Assertion weights.type() == DataType::kINT32 failed.) [12/03/2021-11:47:36] [E] Failed to parse onnx file [12/03/2021-11:47:36] [I] Finish parsing network model [12/03/2021-11:47:36] [E] Parsing model failed [12/03/2021-11:47:36] [E] Failed to create engine from model. [12/03/2021-11:47:36] [E] Engine set up failed &&&& FAILED TensorRT.trtexec [TensorRT v8201] # /workspace/tensorrt/bin/trtexec --onnx=model.onnx --saveEngine=model.engine

[wikipedia2008]

model.onnx in https://drive.google.com/file/d/1Jejuwe3IvB0T1AjftsZeApK-NXHeEBwE/view?usp=sharing

[wikipedia2008]

help

[faithfulnguyen]

I got the same issue as you: Here is the version I use:

• onnx=1.9.0
• tf2onnx=1.8.1
• tensorflow=2.6
• tensorrt=8.2.1.8

I use this command to convert TensorFlow to ONNX:

python3 -m tf2onnx.convert \
 --input  $INPUT_FROZEN_GRAPH_PB \
 --inputs $INPUTS \
 --outputs $OUTPUTS \
 --output $OUTPUT_ONNX_FILE \
 --opset 7 \
 --verbose

I try to change opset value from 7 to 12 and the error (Assertion weights.type() == DataType::kINT32 failed) when convert ONNX to TensorRT is gone.

python3 -m tf2onnx.convert \
 --input  $INPUT_FROZEN_GRAPH_PB \
 --inputs $INPUTS \
 --outputs $OUTPUTS \
 --output $OUTPUT_ONNX_FILE \
 --opset 12 \
 --verbose

hope it helps you.

[zhenhuaw-me]

@wikipedia2008 Thank you for reporting this issue! The log seems related to the limitation of data type w.r.t. dynamic shapes. Could you please try TensorRT 8.4 EA?

[duduscript]

@jackwish Thank you, TensorRT 8.4 EA works for me, but is there any solution in TensorRT 8.2?

[zhenhuaw-me]

@duduscript We don't fix any 8.2 issues anymore. Could you deploy your model with 8.4? And, have you tried the solution mentioned by @faithfulnguyen https://github.com/onnx/onnx-tensorrt/issues/784#issuecomment-1022059925

[duduscript]

Hi @jackwish , I use a pass to avoid dynamic shape in the model I deploy and solved this problem, thanks.

[zhenhuaw-me]

@duduscript Glad to hear that. Closing this issue. Feel free to reopen if any furthur issues.