NVIDIA
polygraphy run error
Description
[08/10/2022-18:18:24] [TRT] [I] Global timing cache in use. Profiling results in this builder pass will be stored. [08/10/2022-18:19:17] [TRT] [W] Skipping tactic 0x0000000000000000 due to Myelin error: Formal output tensor "encoder_cross_views_0_cross_attend_mlp_0_bias _ (Unnamed Layer_ 377) [Shuffle]_constant" is also a data tensor. [08/10/2022-18:19:17] [TRT] [E] 10: [optimizer.cpp::computeCosts::3628] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[encoder.cross_views.0.cross_attend.to_q.1.bias + (Unnamed Layer* 242) [Shuffle]...Reshape_987]}.) [08/10/2022-18:19:17] [TRT] [E] 2: [builder.cpp::buildSerializedNetwork::636] Error Code 2: Internal Error (Assertion engine != nullptr failed. )
Environment
TensorRT Version: 8.4.1.5 NVIDIA GPU: 3090 NVIDIA Driver Version: 470.74 CUDA Version: 11.14 CUDNN Version: 8.2.4 Operating System: ubuntu Python Version (if applicable): 3.8 Tensorflow Version (if applicable): PyTorch Version (if applicable): 1.11 Baremetal or Container (if so, version):
Relevant Files
my model in on [https://github.com/xiongda777/model_file.git]
Steps To Reproduce
hope your help!
I can not reproduce in my test(official docker image nvcr.io/nvidia/tensorrt:22.07-py3 with TRT 8.4):
[08/10/2022-15:02:06] [I] === Performance summary ===
[08/10/2022-15:02:06] [I] Throughput: 44.5772 qps
[08/10/2022-15:02:06] [I] Latency: min = 23.2395 ms, max = 23.3767 ms, mean = 23.3395 ms, median = 23.3398 ms, percentile(99%) = 23.3749 ms
[08/10/2022-15:02:06] [I] Enqueue Time: min = 22.2388 ms, max = 22.4736 ms, mean = 22.3398 ms, median = 22.3413 ms, percentile(99%) = 22.424 ms
[08/10/2022-15:02:06] [I] H2D Latency: min = 0.886841 ms, max = 0.940674 ms, mean = 0.910242 ms, median = 0.910568 ms, percentile(99%) = 0.935547 ms
[08/10/2022-15:02:06] [I] GPU Compute Time: min = 22.2874 ms, max = 22.4213 ms, mean = 22.3863 ms, median = 22.3871 ms, percentile(99%) = 22.4178 ms
[08/10/2022-15:02:06] [I] D2H Latency: min = 0.027832 ms, max = 0.0510254 ms, mean = 0.0429132 ms, median = 0.0429688 ms, percentile(99%) = 0.0471191 ms
[08/10/2022-15:02:06] [I] Total Host Walltime: 3.05089 s
[08/10/2022-15:02:06] [I] Total GPU Compute Time: 3.04454 s
[08/10/2022-15:02:06] [W] * Throughput may be bound by Enqueue Time rather than GPU Compute and the GPU may be under-utilized.
[08/10/2022-15:02:06] [W] If not already in use, --useCudaGraph (utilize CUDA graphs where possible) may increase the throughput.
[08/10/2022-15:02:06] [I] Explanations of the performance metrics are printed in the verbose logs.
[08/10/2022-15:02:06] [I]
&&&& PASSED TensorRT.trtexec [TensorRT v8401] # trtexec --onnx=model.onnx
Do I miss anything?
I can not reproduce in my test(official docker image nvcr.io/nvidia/tensorrt:22.07-py3 with TRT 8.4):
I use 'polygraphy run' to test my model which cause the error,below is my log. However it can be convert to tensorrt engine with trtexec. I don't know why the error be caused.
CUDA_VISIBLE_DEVICES=4 polygraphy run model.onnx --trt --onnxrt --trt-outputs mark all --onnx-outputs mark all --tactic-sources CUBLAS --fp16 --atol 1e-3 --rtol 1e-3 --val-range [0,1] [I] trt-runner-N0-08/11/22-09:46:14 | Activating and starting inference [08/11/2022-09:46:16] [TRT] [W] onnx2trt_utils.cpp:369: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [08/11/2022-09:46:16] [TRT] [W] onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped [I] Configuring with profiles: [Profile().add(image, min=[1, 6, 3, 224, 480], opt=[1, 6, 3, 224, 480], max=[1, 6, 3, 224, 480]).add(intrinsics, min=[1, 6, 3, 3], opt=[1, 6, 3, 3], max=[1, 6, 3, 3]).add(extrinsics, min=[1, 6, 4, 4], opt=[1, 6, 4, 4], max=[1, 6, 4, 4])] [I] Building engine with configuration: Workspace | 16777216 bytes (16.00 MiB) Precision | TF32: False, FP16: True, INT8: False, Obey Precision Constraints: False, Strict Types: False Tactic Sources | ['CUBLAS'] Safety Restricted | False Profiles | 1 profile(s) [08/11/2022-09:47:47] [TRT] [W] Weights [name=Conv_88.weight] had the following issues when converted to FP16: [08/11/2022-09:47:47] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:47] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:47] [TRT] [W] Weights [name=Conv_88.weight] had the following issues when converted to FP16: [08/11/2022-09:47:47] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:47] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:47] [TRT] [W] Weights [name=Conv_88.weight] had the following issues when converted to FP16: [08/11/2022-09:47:47] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:47] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:47] [TRT] [W] Weights [name=Conv_88.weight] had the following issues when converted to FP16: [08/11/2022-09:47:47] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:47] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:47] [TRT] [W] Weights [name=Conv_88.weight] had the following issues when converted to FP16: [08/11/2022-09:47:47] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:47] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:47] [TRT] [W] Weights [name=Conv_88.weight] had the following issues when converted to FP16: [08/11/2022-09:47:47] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:47] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:47] [TRT] [W] Weights [name=Conv_88.weight] had the following issues when converted to FP16: [08/11/2022-09:47:47] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:47] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:47] [TRT] [W] Weights [name=Conv_88.weight] had the following issues when converted to FP16: [08/11/2022-09:47:47] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:47] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:47] [TRT] [W] Weights [name=Conv_88.weight] had the following issues when converted to FP16: [08/11/2022-09:47:47] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:47] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:47] [TRT] [W] Weights [name=Conv_88.weight] had the following issues when converted to FP16: [08/11/2022-09:47:47] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:47] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:47] [TRT] [W] Weights [name=Conv_88.weight] had the following issues when converted to FP16: [08/11/2022-09:47:47] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:47] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:47] [TRT] [W] Weights [name=Conv_88.weight] had the following issues when converted to FP16: [08/11/2022-09:47:47] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:47] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:47] [TRT] [W] Weights [name=Conv_88.weight] had the following issues when converted to FP16: [08/11/2022-09:47:47] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:47] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:47] [TRT] [W] Weights [name=Conv_88.weight] had the following issues when converted to FP16: [08/11/2022-09:47:47] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:47] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:51] [TRT] [W] Weights [name=Conv_110.weight] had the following issues when converted to FP16: [08/11/2022-09:47:51] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:51] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:51] [TRT] [W] Weights [name=Conv_110.weight] had the following issues when converted to FP16: [08/11/2022-09:47:51] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:51] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:51] [TRT] [W] Weights [name=Conv_110.weight] had the following issues when converted to FP16: [08/11/2022-09:47:51] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:51] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:51] [TRT] [W] Weights [name=Conv_110.weight] had the following issues when converted to FP16: [08/11/2022-09:47:51] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:51] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:51] [TRT] [W] Weights [name=Conv_110.weight] had the following issues when converted to FP16: [08/11/2022-09:47:51] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:51] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:51] [TRT] [W] Weights [name=Conv_110.weight] had the following issues when converted to FP16: [08/11/2022-09:47:51] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:51] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:51] [TRT] [W] Weights [name=Conv_110.weight] had the following issues when converted to FP16: [08/11/2022-09:47:51] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:51] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:51] [TRT] [W] Weights [name=Conv_110.weight] had the following issues when converted to FP16: [08/11/2022-09:47:51] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:51] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:51] [TRT] [W] Weights [name=Conv_110.weight] had the following issues when converted to FP16: [08/11/2022-09:47:51] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:51] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:51] [TRT] [W] Weights [name=Conv_110.weight] had the following issues when converted to FP16: [08/11/2022-09:47:51] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:51] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:51] [TRT] [W] Weights [name=Conv_110.weight] had the following issues when converted to FP16: [08/11/2022-09:47:51] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:51] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:51] [TRT] [W] Weights [name=Conv_110.weight] had the following issues when converted to FP16: [08/11/2022-09:47:51] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:51] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:51] [TRT] [W] Weights [name=Conv_110.weight] had the following issues when converted to FP16: [08/11/2022-09:47:51] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:51] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:47:51] [TRT] [W] Weights [name=Conv_110.weight] had the following issues when converted to FP16: [08/11/2022-09:47:51] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:47:51] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=Conv_540.weight] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=Conv_540.weight] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=Conv_540.weight] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=Conv_540.weight] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=Conv_540.weight] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=Conv_540.weight] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=Conv_540.weight] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=Conv_540.weight] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=Conv_540.weight] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=Conv_540.weight] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=Conv_540.weight] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=Conv_540.weight] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=Conv_540.weight] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=Conv_540.weight] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] cuDNN, cuBLAS or cuBLASLt library is still required on networks with loop, boolean operators or transformer based architectures even if it is disabled through TacticSources APIs. [08/11/2022-09:48:36] [TRT] [W] Weights [name=onnx::MatMul_3144 + (Unnamed Layer* 239) [Shuffle]] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=onnx::Add_1240] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=onnx::Add_1147] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=onnx::Add_1469] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=onnx::MatMul_3145 + (Unnamed Layer* 260) [Shuffle]] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=encoder.cross_views.0.cross_attend.to_v.1.bias + (Unnamed Layer* 284) [Shuffle]] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=onnx::MatMul_3146 + (Unnamed Layer* 281) [Shuffle]] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=encoder.cross_views.0.cross_attend.to_v.0.bias + (Unnamed Layer* 278) [Shuffle]] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=encoder.cross_views.0.cross_attend.to_v.0.weight + (Unnamed Layer* 275) [Shuffle]] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=onnx::MatMul_3166 + (Unnamed Layer* 351) [Shuffle]] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=onnx::Add_1728] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=encoder.cross_views.0.cross_attend.prenorm.bias + (Unnamed Layer* 371) [Shuffle]] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=onnx::MatMul_3176 + (Unnamed Layer* 374) [Shuffle]] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Weights [name=onnx::MatMul_3177 + (Unnamed Layer* 391) [Shuffle]] had the following issues when converted to FP16: [08/11/2022-09:48:36] [TRT] [W] - Subnormal FP16 values detected. [08/11/2022-09:48:36] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to reduce the magnitude of the weights. [08/11/2022-09:48:36] [TRT] [W] Skipping tactic 0x0000000000000000 due to Myelin error: Formal output tensor "encoder_cross_views_0_cross_attend_mlp_0_bias _ (Unnamed Layer_ 377) [Shuffle]_constant" is also a data tensor. [08/11/2022-09:48:36] [TRT] [E] 10: [optimizer.cpp::computeCosts::3628] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[encoder.cross_views.0.cross_attend.to_q.1.bias + (Unnamed Layer* 242) [Shuffle]...Reshape_987]}.) [08/11/2022-09:48:36] [TRT] [E] 2: [builder.cpp::buildSerializedNetwork::636] Error Code 2: Internal Error (Assertion engine != nullptr failed. ) [!] Invalid Engine. Please ensure the engine was built correctly
Oops, can you try remove --trt-outputs mark all --onnx-outputs mark all in your command?
Oops, can you try remove
--trt-outputs mark all --onnx-outputs mark allin your command?
It works fine after remove --trt-outputs mark all --onnx-outputs mark all. But I need to compare the accuracy of all the intermediate layers to determine which layer has an accuracy overflow. After I remove mark all, it can only compare the outputs of the output layer. So I want to know how the warning 'Skipping tactic 0x0000000000000000 due to Myelin error: Formal output tensor "encoder_cross_views_0_cross_attend_mlp_0_bias _ (Unnamed Layer_ 377) [Shuffle]_constant" is also a data tensor' is generated and how can I fix it?
We don't recommend setting all intermediate layers as output, it will break TRT's layers fusion and usually affect the output accuracy. the better solution is to check the verbose log of trtexec(trtexec --verbose) and you will find a log section name "Engine Layer Information" when TRT finishes building the engine. it will display the fused layers. set those layers as output so you have the same structure as normal running and it won't break TRT's layer fusion. e.g.:
[08/12/2022-02:31:08] [V] [TRT] Engine Layer Information:
Layer(CaskConvolution): node_of_gpu_0/conv1_1 + node_of_gpu_0/res_conv1_bn_1 + node_of_gpu_0/res_conv1_bn_2, Tactic: 0xc2a5fc6b5e7cef5e, gpu_0/data_0[Float(1,3,224,224)] -> gpu_0/res_conv1_bn_2[Float(1,64,112,112)]
Layer(TiledPooling): node_of_gpu_0/pool1_1, Tactic: 0x00000000006a0101, gpu_0/res_conv1_bn_2[Float(1,64,112,112)] -> gpu_0/pool1_1[Float(1,64,56,56)]
Layer(CaskConvolution): node_of_gpu_0/res2_0_branch2a_1 + node_of_gpu_0/res2_0_branch2a_bn_1 + node_of_gpu_0/res2_0_branch2a_bn_2, Tactic: 0x5193693bc0732c65, gpu_0/pool1_1[Float(1,64,56,56)] -> gpu_0/res2_0_branch2a_bn_2[Float(1,64,56,56)]
Layer(CaskConvolution): node_of_gpu_0/res2_0_branch2b_1 + node_of_gpu_0/res2_0_branch2b_bn_1 + node_of_gpu_0/res2_0_branch2b_bn_2, Tactic: 0x268494f0a1c83de3, gpu_0/res2_0_branch2a_bn_2[Float(1,64,56,56)] -> gpu_0/res2_0_branch2b_bn_2[Float(1,64,56,56)]
Set node_of_gpu_0/res_conv1_bn_2, node_of_gpu_0/pool1_1, node_of_gpu_0/res2_0_branch2a_bn_2 and node_of_gpu_0/res2_0_branch2b_bn_2 as output is best choice other than set every intermidiate tensors as output.
@zerollzeng Thank you for your reply. But after converting to trt, the network structure seems to be reorganized, and the layer name has changed, how can I compare the output of the corresponding onnx layer and the trt layer (the output layer of onnx is more than the output layer of trt, because trt fuses the layers of onnx)?
The layers name will remain unchanged in the engine, e.g.
Layer(CaskConvolution): node_of_gpu_0/conv1_1 + node_of_gpu_0/res_conv1_bn_1 + node_of_gpu_0/res_conv1_bn_2, Tactic: 0xc2a5fc6b5e7cef5e, gpu_0/data_0[Float(1,3,224,224)] -> gpu_0/res_conv1_bn_2[Float(1,64,112,112)]
Layer(CaskConvolution): A + B + C, A,B and C are all onnx layer names, you can find them in the onnx model. if you see some names like unnamed xxx, just ignore it, they are produced by TRT for data reformat or other purposes.
closing since no activity for more than 3 weeks, please reopen if you still have question, thanks!
