deepvac
deepvac copied to clipboard
tensorrt转换器报错
当打开tensorrt转换器开关后,转换逻辑报错。
Description
[TensorRT] ERROR: Try increasing the workspace size with IBuilderConfig::setMaxWorkspaceSize() if using IBuilder::buildEngineWithConfig, or IBuilder::setMaxWorkspaceSize() if using IBuilder::buildCudaEngine.
[TensorRT] ERROR: ../builder/tacticOptimizer.cpp (1820) - TRTInternal Error in computeCosts: 0 (Could not find any implementation for node 2-layer MLP: Gemm_56 + Relu_57 -> Gemm_58 + 544 + (Unnamed Layer* 84) [Shuffle] + unsqueeze_node_after_544 + (Unnamed Layer* 84) [Shuffle] + Add_60.)
[TensorRT] ERROR: ../builder/tacticOptimizer.cpp (1820) - TRTInternal Error in computeCosts: 0 (Could not find any implementation for node 2-layer MLP: Gemm_56 + Relu_57 -> Gemm_58 + 544 + (Unnamed Layer* 84) [Shuffle] + unsqueeze_node_after_544 + (Unnamed Layer* 84) [Shuffle] + Add_60.)
I face the problem with 1080 and no problem on 2080
Environment
Ubuntu 20.04 + MLab HomePod 1.1
it seems related with this issue: https://github.com/NVIDIA/TensorRT/issues/851
I enable verbose log to see the detail failure.I find some difference between 1080 and 2080.
1080 log
[TensorRT] VERBOSE: --------------- Timing Runner: GlobalAveragePool_51 (Pooling)
[TensorRT] VERBOSE: Tactic: -1 time 0.031744
[TensorRT] VERBOSE: Fastest Tactic: -1 Time: 0.031744
[TensorRT] VERBOSE: --------------- Timing Runner: GlobalAveragePool_51 (TiledPooling)
[TensorRT] VERBOSE: TiledPooling has no valid tactics for this config, skipping
[TensorRT] VERBOSE: >>>>>>>>>>>>>>> Chose Runner Type: Pooling Tactic: -1
[TensorRT] VERBOSE:
[TensorRT] VERBOSE: *************** Autotuning format combination: Float(1,1,1,72) -> Float(1,1,1,72) ***************
[TensorRT] VERBOSE: --------------- Timing Runner: 2-layer MLP: Gemm_56 + Relu_57 -> Gemm_58 + 544 + (Unnamed Layer* 84) [Shuffle] + unsqueeze_node_after_544 + (Unnamed Layer* 84) [Shuffle] + Add_60 (CudnnMLPFC)
[TensorRT] VERBOSE: Tactic: 0 skipped. Scratch requested: 192, available: 0
[TensorRT] VERBOSE: Tactic: 1 skipped. Scratch requested: 192, available: 0
[TensorRT] VERBOSE: Fastest Tactic: -3360065831133338131 Time: 3.40282e+38
[TensorRT] VERBOSE: --------------- Timing Runner: 2-layer MLP: Gemm_56 + Relu_57 -> Gemm_58 + 544 + (Unnamed Layer* 84) [Shuffle] + unsqueeze_node_after_544 + (Unnamed Layer* 84) [Shuffle] + Add_60 (CudnnMLPMM)
[TensorRT] VERBOSE: Tactic: 0 skipped. Scratch requested: 192, available: 0
[TensorRT] VERBOSE: Fastest Tactic: -3360065831133338131 Time: 3.40282e+38
[TensorRT] VERBOSE: --------------- Timing Runner: 2-layer MLP: Gemm_56 + Relu_57 -> Gemm_58 + 544 + (Unnamed Layer* 84) [Shuffle] + unsqueeze_node_after_544 + (Unnamed Layer* 84) [Shuffle] + Add_60 (CudaMLP)
[TensorRT] VERBOSE: CudaMLP has no valid tactics for this config, skipping
[TensorRT] ERROR: Try increasing the workspace size with IBuilderConfig::setMaxWorkspaceSize() if using IBuilder::buildEngineWithConfig, or IBuilder::setMaxWorkspaceSize() if using IBuilder::buildCudaEngine.
[TensorRT] ERROR: ../builder/tacticOptimizer.cpp (1820) - TRTInternal Error in computeCosts: 0 (Could not find any implementation for node 2-layer MLP: Gemm_56 + Relu_57 -> Gemm_58 + 544 + (Unnamed Layer* 84) [Shuffle] + unsqueeze_node_after_544 + (Unnamed Layer* 84) [Shuffle] + Add_60.)
[TensorRT] VERBOSE: Builder timing cache: created 62 entries, 41 hit(s)
[TensorRT] ERROR: ../builder/tacticOptimizer.cpp (1820) - TRTInternal Error in computeCosts: 0 (Could not find any implementation for node 2-layer MLP: Gemm_56 + Relu_57 -> Gemm_58 + 544 + (Unnamed Layer* 84) [Shuffle] + unsqueeze_node_after_544 + (Unnamed Layer* 84) [Shuffle] + Add_60.)
2080 log
[TensorRT] VERBOSE: --------------- Timing Runner: GlobalAveragePool_51 (TiledPooling)
[TensorRT] VERBOSE: TiledPooling has no valid tactics for this config, skipping
[TensorRT] VERBOSE: >>>>>>>>>>>>>>> Chose Runner Type: Pooling Tactic: -1
[TensorRT] VERBOSE:
[TensorRT] VERBOSE: *************** Autotuning format combination: Float(1,1,1,72) -> Float(1,1,1,72) ***************
[TensorRT] VERBOSE: --------------- Timing Runner: 2-layer MLP: Gemm_56 + Relu_57 -> Gemm_58 + 544 + (Unnamed Layer* 84) [Shuffle] + unsqueeze_node_after_544 + (Unnamed Layer* 84) [Shuffle] + Add_60 (CudnnMLPFC)
[TensorRT] VERBOSE: Tactic: 0 skipped. Scratch requested: 192, available: 0
[TensorRT] VERBOSE: Tactic: 1 skipped. Scratch requested: 192, available: 0
[TensorRT] VERBOSE: Fastest Tactic: -3360065831133338131 Time: 3.40282e+38
[TensorRT] VERBOSE: --------------- Timing Runner: 2-layer MLP: Gemm_56 + Relu_57 -> Gemm_58 + 544 + (Unnamed Layer* 84) [Shuffle] + unsqueeze_node_after_544 + (Unnamed Layer* 84) [Shuffle] + Add_60 (CudnnMLPMM)
[TensorRT] VERBOSE: Tactic: 0 skipped. Scratch requested: 192, available: 0
[TensorRT] VERBOSE: Fastest Tactic: -3360065831133338131 Time: 3.40282e+38
[TensorRT] VERBOSE: --------------- Timing Runner: 2-layer MLP: Gemm_56 + Relu_57 -> Gemm_58 + 544 + (Unnamed Layer* 84) [Shuffle] + unsqueeze_node_after_544 + (Unnamed Layer* 84) [Shuffle] + Add_60 (CudaMLP)
[TensorRT] VERBOSE: Tactic: 0 time 0.01024
[TensorRT] VERBOSE: Fastest Tactic: 0 Time: 0.01024
[TensorRT] VERBOSE: >>>>>>>>>>>>>>> Chose Runner Type: CudaMLP Tactic: 0
I only find "CudaMLP has no valid tactics for this config skipping" in 1080's log. Maybe this caused some error occuerd on 1080 but not error on 2080.