"Found duplicate region name" error when load onnx model

[WhatGhost]

Description

i meet a error when i run trtexec to load my model i first trans my model from paddle2onnx and then use trtexec to load the onnx model and meet this error "Found duplicate region name p2o.Mul.12'[shuffle input])" it seems like it has two same node .But i cant find it using netron. the p2o.Mul.12 is the output of a Mul node and be used by two difference op.

Please help me ,how can i solve this problem! Thanks very much!!!

[09/22/2022-20:32:50] [W] Explicit batch dim is automatically enabled if input model is ONNX or if dynamic shapes are provided when the engine is built. [09/22/2022-20:32:50] [I] === Model Options === [09/22/2022-20:32:50] [I] Format: ONNX [09/22/2022-20:32:50] [I] Model: er.onnx [09/22/2022-20:32:50] [I] Output: [09/22/2022-20:32:50] [I] === Build Options === [09/22/2022-20:32:50] [I] Max batch: explicit batch [09/22/2022-20:32:50] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default [09/22/2022-20:32:50] [I] minTiming: 1 [09/22/2022-20:32:50] [I] avgTiming: 8 [09/22/2022-20:32:50] [I] Precision: FP32+FP16 [09/22/2022-20:32:50] [I] LayerPrecisions: [09/22/2022-20:32:50] [I] Calibration: [09/22/2022-20:32:50] [I] Refit: Disabled [09/22/2022-20:32:50] [I] Sparsity: Disabled [09/22/2022-20:32:50] [I] Safe mode: Disabled [09/22/2022-20:32:50] [I] DirectIO mode: Disabled [09/22/2022-20:32:50] [I] Restricted mode: Disabled [09/22/2022-20:32:50] [I] Save engine: ernie_rel_model.trt [09/22/2022-20:32:50] [I] Load engine: [09/22/2022-20:32:50] [I] Profiling verbosity: 0 [09/22/2022-20:32:50] [I] Tactic sources: Using default tactic sources [09/22/2022-20:32:50] [I] timingCacheMode: local [09/22/2022-20:32:50] [I] timingCacheFile: [09/22/2022-20:32:50] [I] Input(s)s format: fp32:CHW [09/22/2022-20:32:50] [I] Output(s)s format: fp32:CHW [09/22/2022-20:32:50] [I] Input build shape: sids=1x128+10x128+10x128 [09/22/2022-20:32:50] [I] Input build shape: ids=1x128+10x128+10x128 [09/22/2022-20:32:50] [I] Input calibration shapes: model [09/22/2022-20:32:50] [I] === System Options === [09/22/2022-20:32:50] [I] Device: 0 [09/22/2022-20:32:50] [I] DLACore: [09/22/2022-20:32:50] [I] Plugins: [09/22/2022-20:32:50] [I] === Inference Options === [09/22/2022-20:32:50] [I] Batch: Explicit [09/22/2022-20:32:50] [I] Input inference shape: ids=10x128 [09/22/2022-20:32:50] [I] Input inference shape: sids=10x128 [09/22/2022-20:32:50] [I] Iterations: 10 [09/22/2022-20:32:50] [I] Duration: 3s (+ 200ms warm up) [09/22/2022-20:32:50] [I] Sleep time: 0ms [09/22/2022-20:32:50] [I] Idle time: 0ms [09/22/2022-20:32:50] [I] Streams: 1 [09/22/2022-20:32:50] [I] ExposeDMA: Disabled [09/22/2022-20:32:50] [I] Data transfers: Enabled [09/22/2022-20:32:50] [I] Spin-wait: Disabled [09/22/2022-20:32:50] [I] Multithreading: Disabled [09/22/2022-20:32:50] [I] CUDA Graph: Disabled [09/22/2022-20:32:50] [I] Separate profiling: Disabled [09/22/2022-20:32:50] [I] Time Deserialize: Disabled [09/22/2022-20:32:50] [I] Time Refit: Disabled [09/22/2022-20:32:50] [I] Skip inference: Disabled [09/22/2022-20:32:50] [I] Inputs: [09/22/2022-20:32:50] [I] === Reporting Options === [09/22/2022-20:32:50] [I] Verbose: Disabled [09/22/2022-20:32:50] [I] Averages: 10 inferences [09/22/2022-20:32:50] [I] Percentile: 99 [09/22/2022-20:32:50] [I] Dump refittable layers:Disabled [09/22/2022-20:32:50] [I] Dump output: Disabled [09/22/2022-20:32:50] [I] Profile: Disabled [09/22/2022-20:32:50] [I] Export timing to JSON file: [09/22/2022-20:32:50] [I] Export output to JSON file: [09/22/2022-20:32:50] [I] Export profile to JSON file: [09/22/2022-20:32:50] [I] [09/22/2022-20:32:50] [I] === Device Information === [09/22/2022-20:32:50] [I] Selected Device: NVIDIA A10 [09/22/2022-20:32:50] [I] Compute Capability: 8.6 [09/22/2022-20:32:50] [I] SMs: 72 [09/22/2022-20:32:50] [I] Compute Clock Rate: 1.695 GHz [09/22/2022-20:32:50] [I] Device Global Memory: 22731 MiB [09/22/2022-20:32:50] [I] Shared Memory per SM: 100 KiB [09/22/2022-20:32:50] [I] Memory Bus Width: 384 bits (ECC enabled) [09/22/2022-20:32:50] [I] Memory Clock Rate: 6.251 GHz [09/22/2022-20:32:50] [I] [09/22/2022-20:32:50] [I] TensorRT version: 8.4.0 [09/22/2022-20:32:51] [I] [TRT] [MemUsageChange] Init CUDA: CPU +225, GPU +0, now: CPU 233, GPU 16121 (MiB) [09/22/2022-20:32:52] [I] [TRT] [MemUsageSnapshot] Begin constructing builder kernel library: CPU 4294961228 MiB, GPU 16121 MiB [09/22/2022-20:32:52] [I] [TRT] [MemUsageSnapshot] End constructing builder kernel library: CPU 3892345657 MiB, GPU 16243 MiB [09/22/2022-20:32:52] [I] Start parsing network model [09/22/2022-20:32:52] [I] [TRT] ---------------------------------------------------------------- [09/22/2022-20:32:52] [I] [TRT] Input filename: er.onnx [09/22/2022-20:32:52] [I] [TRT] ONNX IR version: 0.0.8 [09/22/2022-20:32:52] [I] [TRT] Opset version: 11 [09/22/2022-20:32:52] [I] [TRT] Producer name: [09/22/2022-20:32:52] [I] [TRT] Producer version: [09/22/2022-20:32:52] [I] [TRT] Domain: [09/22/2022-20:32:52] [I] [TRT] Model version: 0 [09/22/2022-20:32:52] [I] [TRT] Doc string: [09/22/2022-20:32:52] [I] [TRT] ---------------------------------------------------------------- [09/22/2022-20:32:53] [W] [TRT] onnx2trt_utils.cpp:365: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [09/22/2022-20:33:09] [I] Finish parsing network model [09/22/2022-20:33:14] [W] [TRT] TensorRT was linked against cuBLAS/cuBLAS LT 11.8.0 but loaded cuBLAS/cuBLAS LT 11.2.1 [09/22/2022-20:33:14] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +621, GPU +260, now: CPU 1879, GPU 16503 (MiB) [09/22/2022-20:33:15] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +641, GPU +266, now: CPU 2518, GPU 16769 (MiB) [09/22/2022-20:33:15] [W] [TRT] TensorRT was linked against cuDNN 8.3.2 but loaded cuDNN 8.1.1 [09/22/2022-20:33:15] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored. [09/22/2022-20:33:15] [E] Error[2]: [checkSanity.cpp::checkSanity::106] Error Code 2: Internal Error (Assertion regionNames.find(r->name) == regionNames.end() failed. Found duplicate region name p2o.Mul.12'[shuffle input]) [09/22/2022-20:33:15] [E] Error[2]: [builder.cpp::buildSerializedNetwork::619] Error Code 2: Internal Error (Assertion engine != nullptr failed. ) [09/22/2022-20:33:15] [E] Engine could not be created from network [09/22/2022-20:33:15] [E] Building engine failed [09/22/2022-20:33:15] [E] Failed to create engine from model. [09/22/2022-20:33:15] [E] Engine set up failed

Environment

TensorRT Version: 8.4.0.6 NVIDIA GPU: A10 NVIDIA Driver Version:470.82.01 CUDA Version: cuda 11.1 CUDNN Version: 8.1.1 Operating System: Centos 7.5.1804 Python Version (if applicable): Tensorflow Version (if applicable): PyTorch Version (if applicable): Baremetal or Container (if so, version):

Relevant Files

Steps To Reproduce

[zerollzeng]

I think you can use polygraphy to debug this,

1. use polygraphy inspect model model.onnx --show layers attrs | grep p2o.Mul.12 to check if there are duplicate node or tensor.

2. use polygraphy surgeon sanitize model.onnx --fold-constants -o model_foled.onnx might be helpful

if the above methods won't work can you share the onnx here?

[WhatGhost]

Thanks for your reply , i'will try it

[WhatGhost]

i have try these two method . i cant find duplicate node or tensor.using polygraphy. And it still has the same error afert using fold-constants

But due to confidentiality requirementsd,i cant share the onnx model . Thanks very much for your reply!

[asmaaEzz]

any updates regarding this? @WhatGhost

[asmaaEzz]

Have you tried upgrading trt version to 8.4.1 ?

[ttyio]

I will close inactive issues for more than 3 week per our policy, thanks all!