TensorRT icon indicating copy to clipboard operation
TensorRT copied to clipboard

Published 20 hours ago • verified publisher icon NVIDIA

3D asymmetric padding is not supported on pre-sm70 GPUs

Open Ri-Yue-Shui-Hua opened this issue 2 years ago • 5 comments

Description

[E] Error[10]: ConvTranspose_19: 3D asymmetric padding is not supported on pre-sm70 GPUs [06/26/2022-18:34:14] [E] [TRT] ModelImporter.cpp:773: While parsing node number 19 [ConvTranspose -> "134"]:

Environment

TensorRT Version: 8.4.1.5 NVIDIA GPU: NVIDIA GeForce1080Ti NVIDIA Driver Version: 470.129.06 CUDA Version: 11.4 CUDNN Version: 8.4.1 Operating System: Ubuntu18.04 Python Version (if applicable): 3.8 Tensorflow Version (if applicable): PyTorch Version (if applicable): 1.11 Baremetal or Container (if so, version):

Relevant Files

链接: https://pan.baidu.com/s/1vODujQlZ8dz1CXEft8qnWw 密码: r1bl --来自百度网盘超级会员V5的分享

Steps To Reproduce

 ./trtexec --onnx=model.onnx --saveEngine=model.engine
&&&& RUNNING TensorRT.trtexec [TensorRT v8401] # ./trtexec --onnx=model.onnx --saveEngine=model.engine
[06/26/2022-18:34:14] [I] === Model Options ===
[06/26/2022-18:34:14] [I] Format: ONNX
[06/26/2022-18:34:14] [I] Model: model.onnx
[06/26/2022-18:34:14] [I] Output:
[06/26/2022-18:34:14] [I] === Build Options ===
[06/26/2022-18:34:14] [I] Max batch: explicit batch
[06/26/2022-18:34:14] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[06/26/2022-18:34:14] [I] minTiming: 1
[06/26/2022-18:34:14] [I] avgTiming: 8
[06/26/2022-18:34:14] [I] Precision: FP32
[06/26/2022-18:34:14] [I] LayerPrecisions: 
[06/26/2022-18:34:14] [I] Calibration: 
[06/26/2022-18:34:14] [I] Refit: Disabled
[06/26/2022-18:34:14] [I] Sparsity: Disabled
[06/26/2022-18:34:14] [I] Safe mode: Disabled
[06/26/2022-18:34:14] [I] DirectIO mode: Disabled
[06/26/2022-18:34:14] [I] Restricted mode: Disabled
[06/26/2022-18:34:14] [I] Build only: Disabled
[06/26/2022-18:34:14] [I] Save engine: model.engine
[06/26/2022-18:34:14] [I] Load engine: 
[06/26/2022-18:34:14] [I] Profiling verbosity: 0
[06/26/2022-18:34:14] [I] Tactic sources: Using default tactic sources
[06/26/2022-18:34:14] [I] timingCacheMode: local
[06/26/2022-18:34:14] [I] timingCacheFile: 
[06/26/2022-18:34:14] [I] Input(s)s format: fp32:CHW
[06/26/2022-18:34:14] [I] Output(s)s format: fp32:CHW
[06/26/2022-18:34:14] [I] Input build shapes: model
[06/26/2022-18:34:14] [I] Input calibration shapes: model
[06/26/2022-18:34:14] [I] === System Options ===
[06/26/2022-18:34:14] [I] Device: 0
[06/26/2022-18:34:14] [I] DLACore: 
[06/26/2022-18:34:14] [I] Plugins:
[06/26/2022-18:34:14] [I] === Inference Options ===
[06/26/2022-18:34:14] [I] Batch: Explicit
[06/26/2022-18:34:14] [I] Input inference shapes: model
[06/26/2022-18:34:14] [I] Iterations: 10
[06/26/2022-18:34:14] [I] Duration: 3s (+ 200ms warm up)
[06/26/2022-18:34:14] [I] Sleep time: 0ms
[06/26/2022-18:34:14] [I] Idle time: 0ms
[06/26/2022-18:34:14] [I] Streams: 1
[06/26/2022-18:34:14] [I] ExposeDMA: Disabled
[06/26/2022-18:34:14] [I] Data transfers: Enabled
[06/26/2022-18:34:14] [I] Spin-wait: Disabled
[06/26/2022-18:34:14] [I] Multithreading: Disabled
[06/26/2022-18:34:14] [I] CUDA Graph: Disabled
[06/26/2022-18:34:14] [I] Separate profiling: Disabled
[06/26/2022-18:34:14] [I] Time Deserialize: Disabled
[06/26/2022-18:34:14] [I] Time Refit: Disabled
[06/26/2022-18:34:14] [I] Inputs:
[06/26/2022-18:34:14] [I] === Reporting Options ===
[06/26/2022-18:34:14] [I] Verbose: Disabled
[06/26/2022-18:34:14] [I] Averages: 10 inferences
[06/26/2022-18:34:14] [I] Percentile: 99
[06/26/2022-18:34:14] [I] Dump refittable layers:Disabled
[06/26/2022-18:34:14] [I] Dump output: Disabled
[06/26/2022-18:34:14] [I] Profile: Disabled
[06/26/2022-18:34:14] [I] Export timing to JSON file: 
[06/26/2022-18:34:14] [I] Export output to JSON file: 
[06/26/2022-18:34:14] [I] Export profile to JSON file: 
[06/26/2022-18:34:14] [I] 
[06/26/2022-18:34:14] [I] === Device Information ===
[06/26/2022-18:34:14] [I] Selected Device: NVIDIA GeForce GTX 1080 Ti
[06/26/2022-18:34:14] [I] Compute Capability: 6.1
[06/26/2022-18:34:14] [I] SMs: 28
[06/26/2022-18:34:14] [I] Compute Clock Rate: 1.582 GHz
[06/26/2022-18:34:14] [I] Device Global Memory: 11175 MiB
[06/26/2022-18:34:14] [I] Shared Memory per SM: 96 KiB
[06/26/2022-18:34:14] [I] Memory Bus Width: 352 bits (ECC disabled)
[06/26/2022-18:34:14] [I] Memory Clock Rate: 5.505 GHz
[06/26/2022-18:34:14] [I] 
[06/26/2022-18:34:14] [I] TensorRT version: 8.4.1
[06/26/2022-18:34:14] [I] [TRT] [MemUsageChange] Init CUDA: CPU +196, GPU +0, now: CPU 204, GPU 615 (MiB)
[06/26/2022-18:34:14] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +7, GPU +2, now: CPU 228, GPU 617 (MiB)
[06/26/2022-18:34:14] [I] Start parsing network model
[06/26/2022-18:34:14] [I] [TRT] ----------------------------------------------------------------
[06/26/2022-18:34:14] [I] [TRT] Input filename:   modelg.onnx
[06/26/2022-18:34:14] [I] [TRT] ONNX IR version:  0.0.7
[06/26/2022-18:34:14] [I] [TRT] Opset version:    12
[06/26/2022-18:34:14] [I] [TRT] Producer name:    pytorch
[06/26/2022-18:34:14] [I] [TRT] Producer version: 1.10
[06/26/2022-18:34:14] [I] [TRT] Domain:           
[06/26/2022-18:34:14] [I] [TRT] Model version:    0
[06/26/2022-18:34:14] [I] [TRT] Doc string:       
[06/26/2022-18:34:14] [I] [TRT] ----------------------------------------------------------------
[06/26/2022-18:34:14] [E] Error[10]: ConvTranspose_19: 3D asymmetric padding is not supported on pre-sm70 GPUs
[06/26/2022-18:34:14] [E] [TRT] ModelImporter.cpp:773: While parsing node number 19 [ConvTranspose -> "134"]:
[06/26/2022-18:34:14] [E] [TRT] ModelImporter.cpp:774: --- Begin node ---
[06/26/2022-18:34:14] [E] [TRT] ModelImporter.cpp:775: input: "133"
input: "decoder_3.trans_conv.weight"
input: "decoder_3.trans_conv.bias"
output: "134"
name: "ConvTranspose_19"
op_type: "ConvTranspose"
attribute {
  name: "dilations"
  ints: 1
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "group"
  i: 1
  type: INT
}
attribute {
  name: "kernel_shape"
  ints: 3
  ints: 3
  ints: 3
  type: INTS
}
attribute {
  name: "output_padding"
  ints: 1
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "pads"
  ints: 1
  ints: 1
  ints: 1
  ints: 1
  ints: 1
  ints: 1
  type: INTS
}
attribute {
  name: "strides"
  ints: 2
  ints: 2
  ints: 2
  type: INTS
}

[06/26/2022-18:34:14] [E] [TRT] ModelImporter.cpp:776: --- End node ---
[06/26/2022-18:34:14] [E] [TRT] ModelImporter.cpp:779: ERROR: ModelImporter.cpp:167 In function parseGraph:
[6] Invalid Node - ConvTranspose_19
ConvTranspose_19: 3D asymmetric padding is not supported on pre-sm70 GPUs
[06/26/2022-18:34:14] [E] Failed to parse onnx file
[06/26/2022-18:34:14] [I] Finish parsing network model
[06/26/2022-18:34:14] [E] Parsing model failed
[06/26/2022-18:34:14] [E] Failed to create engine from model or file.
[06/26/2022-18:34:14] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8401] # ./trtexec --onnx=model.onnx --saveEngine=model.engine

Ri-Yue-Shui-Hua avatar Jun 26 '22 11:06 Ri-Yue-Shui-Hua

Looks like a device compatibility issue, did you try it on a new device like 2080(sm75+)? refer to https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/

zerollzeng avatar Jun 28 '22 02:06 zerollzeng

Looks like a device compatibility issue, did you try it on a new device like 2080(sm75+)? refer to https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/

appreciate for your reply. I have tried on device such as 1650Ti and 3090 and converted the model successfully. I hope to convert it on device like 1080Ti， how can I do or what should I do?

Ri-Yue-Shui-Hua avatar Jun 28 '22 02:06 Ri-Yue-Shui-Hua

3D asymmetric padding is not supported on pre-sm70 GPUs usually means we don't have kernels for your GPU(1080Ti). @nvpohanh correct me if I'm wrong

zerollzeng avatar Jun 28 '22 03:06 zerollzeng

correct, we don't have tactics for 3d convs with asymmetric padding on old GPUs like Pascal GPUs.

nvpohanh avatar Jun 30 '22 10:06 nvpohanh

TensorRT Version: 8.4.1.5 NVIDIA GPU: NVIDIA GeForce1080 NVIDIA Driver Version: 456.71 CUDA Version: 11.1 CUDNN Version: 8.0.4 Operating System: win10 Python Version (if applicable): 3.7 PyTorch Version (if applicable): 1.11

ERROR: [08/03/2022-14:34:05] [E] [TRT] ModelImporter.cpp:776: --- End node --- [08/03/2022-14:34:05] [E] [TRT] ModelImporter.cpp:779: ERROR: ModelImporter.cpp:167 In function parseGraph: [6] Invalid Node - ConvTranspose_15 ConvTranspose_15: 3D asymmetric padding is not supported on pre-sm70 GPUs [08/03/2022-14:34:05] [E] Failed to parse onnx file [08/03/2022-14:34:05] [I] Finish parsing network model [08/03/2022-14:34:05] [E] Parsing model failed [08/03/2022-14:34:05] [E] Failed to create engine from model or file. [08/03/2022-14:34:05] [E] Engine set up failed

I have the same problem as you. Have you solved it? I am looking forward to your help.

num-doc avatar Aug 03 '22 06:08 num-doc

closing since no activity for more than 14 days, please reopen if you still have question, thanks!

ttyio avatar Dec 12 '22 07:12 ttyio