mmdeploy icon indicating copy to clipboard operation
mmdeploy copied to clipboard

Published 20 hours ago • verified publisher icon open-mmlab

[Bug] AssertionError: Failed to create TensorRT engine

Open TouchDeeper opened this issue 8 months ago • 4 comments

Checklist

  • [X] I have searched related issues but cannot get the expected help.
  • [X] 2. I have read the FAQ documentation but cannot get the expected help.
  • [X] 3. The bug has not been fixed in the latest version.

Describe the bug

AssertionError: Failed to create TensorRT engine

Reproduction

According to the convert-model tutorial, I run the command below:

python mmdeploy/tools/deploy.py \
    mmdeploy/configs/mmdet/detection/detection_tensorrt_dynamic-320x320-1344x1344.py \
    mmdetection/configs/faster_rcnn/faster-rcnn_r50_fpn_1x_coco.py \
    checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
    mmdetection/demo/demo.jpg \
    --work-dir mmdeploy_model/faster-rcnn \
    --device cuda:7 \
    --dump-info

the output log is below:

10/09 15:59:11 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "Codebases" registry tree. As a workaround, the current "Codebases" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
10/09 15:59:11 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "mmdet_tasks" registry tree. As a workaround, the current "mmdet_tasks" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
10/09 15:59:12 - mmengine - INFO - Start pipeline mmdeploy.apis.pytorch2onnx.torch2onnx in subprocess
10/09 15:59:13 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "Codebases" registry tree. As a workaround, the current "Codebases" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
10/09 15:59:13 - mmengine - WARNING - Failed to search registry with scope "mmdet" in the "mmdet_tasks" registry tree. As a workaround, the current "mmdet_tasks" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet" is a correct scope, or whether the registry is initialized.
Loads checkpoint by local backend from path: checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
10/09 15:59:17 - mmengine - WARNING - DeprecationWarning: get_onnx_config will be deprecated in the future. 
10/09 15:59:17 - mmengine - INFO - Export PyTorch model to ONNX: mmdeploy_model/faster-rcnn/end2end.onnx.
10/09 15:59:17 - mmengine - WARNING - Can not find torch.nn.functional.scaled_dot_product_attention, function rewrite will not be applied
10/09 15:59:17 - mmengine - WARNING - Can not find torch._C._jit_pass_onnx_autograd_function_process, function rewrite will not be applied
10/09 15:59:17 - mmengine - WARNING - Can not find mmdet.models.utils.transformer.PatchMerging.forward, function rewrite will not be applied
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVMultiLevelRoiAlign type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVMultiLevelRoiAlign type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::GatherTopk type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::MMCVMultiLevelRoiAlign type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
10/09 15:59:30 - mmengine - INFO - Execute onnx optimize passes.
10/09 15:59:30 - mmengine - WARNING - Can not optimize model, please build torchscipt extension.
More details: https://github.com/open-mmlab/mmdeploy/tree/main/docs/en/experimental/onnx_optimizer.md
10/09 15:59:32 - mmengine - INFO - Finish pipeline mmdeploy.apis.pytorch2onnx.torch2onnx
/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/core/optimizers/function_marker.py:160: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  ys_shape = tuple(int(s) for s in ys.shape)
/data2/home2/admin/ztd_ws/mmdetection/mmdet/models/dense_heads/anchor_head.py:115: UserWarning: DeprecationWarning: anchor_generator is deprecated, please use "prior_generator" instead
  warnings.warn('DeprecationWarning: anchor_generator is deprecated, '
/data2/home2/admin/ztd_ws/mmdetection/mmdet/models/task_modules/prior_generators/anchor_generator.py:356: UserWarning: ``grid_anchors`` would be deprecated soon. Please use ``grid_priors`` 
  warnings.warn('``grid_anchors`` would be deprecated soon. '
/data2/home2/admin/ztd_ws/mmdetection/mmdet/models/task_modules/prior_generators/anchor_generator.py:392: UserWarning: ``single_level_grid_anchors`` would be deprecated soon. Please use ``single_level_grid_priors`` 
  warnings.warn(
/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/codebase/mmdet/models/dense_heads/rpn_head.py:89: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert cls_score.size()[-2:] == bbox_pred.size()[-2:]
/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/pytorch/functions/topk.py:58: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if k > size:
/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/codebase/mmdet/models/task_modules/coders/delta_xywh_bbox_coder.py:38: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert pred_bboxes.size(0) == bboxes.size(0)
/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/codebase/mmdet/models/task_modules/coders/delta_xywh_bbox_coder.py:40: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert pred_bboxes.size(1) == bboxes.size(1)
/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/codebase/mmdet/deploy/utils.py:95: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  assert len(max_shape) == 2, '`max_shape` should be [h, w]'
/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/mmcv/ops/nms.py:474: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  int(scores.shape[-1]),
/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/mmcv/ops/nms.py:148: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  out_boxes = min(num_boxes, after_topk)
/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/codebase/mmdet/models/roi_heads/standard_roi_head.py:41: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  rois_dims = int(rois.shape[-1])
/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/torch/onnx/symbolic_opset9.py:4189: UserWarning: Exporting aten::index operator of advanced indexing in opset 11 is achieved by combination of multiple ONNX operators, including Reshape, Transpose, Concat, and Gather. If indices include negative values, the exported graph will produce incorrect results.
  warnings.warn(
10/09 15:59:34 - mmengine - INFO - Start pipeline mmdeploy.apis.utils.utils.to_backend in subprocess
10/09 15:59:35 - mmengine - INFO - Successfully loaded tensorrt plugins from /home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/lib/libmmdeploy_tensorrt_ops.so
[10/09/2023-15:59:35] [TRT] [I] [MemUsageChange] Init CUDA: CPU +314, GPU +0, now: CPU 398, GPU 10957 (MiB)
[10/09/2023-15:59:35] [TRT] [I] [MemUsageSnapshot] Begin constructing builder kernel library: CPU 398 MiB, GPU 10957 MiB
[10/09/2023-15:59:36] [TRT] [I] [MemUsageSnapshot] End constructing builder kernel library: CPU 533 MiB, GPU 10991 MiB
[10/09/2023-15:59:36] [TRT] [I] ----------------------------------------------------------------
[10/09/2023-15:59:36] [TRT] [I] Input filename:   mmdeploy_model/faster-rcnn/end2end.onnx
[10/09/2023-15:59:36] [TRT] [I] ONNX IR version:  0.0.6
[10/09/2023-15:59:36] [TRT] [I] Opset version:    11
[10/09/2023-15:59:36] [TRT] [I] Producer name:    pytorch
[10/09/2023-15:59:36] [TRT] [I] Producer version: 1.12.1
[10/09/2023-15:59:36] [TRT] [I] Domain:           
[10/09/2023-15:59:36] [TRT] [I] Model version:    0
[10/09/2023-15:59:36] [TRT] [I] Doc string:       
[10/09/2023-15:59:36] [TRT] [I] ----------------------------------------------------------------
[10/09/2023-15:59:36] [TRT] [W] onnx2trt_utils.cpp:366: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[10/09/2023-15:59:36] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[10/09/2023-15:59:37] [TRT] [I] No importer registered for op: GatherTopk. Attempting to import as plugin.
[10/09/2023-15:59:37] [TRT] [I] Searching for plugin: GatherTopk, plugin_version: 1, plugin_namespace: 
[10/09/2023-15:59:37] [TRT] [I] Successfully created plugin: GatherTopk
[10/09/2023-15:59:37] [TRT] [I] No importer registered for op: GatherTopk. Attempting to import as plugin.
[10/09/2023-15:59:37] [TRT] [I] Searching for plugin: GatherTopk, plugin_version: 1, plugin_namespace: 
[10/09/2023-15:59:37] [TRT] [I] Successfully created plugin: GatherTopk
[10/09/2023-15:59:37] [TRT] [I] No importer registered for op: GatherTopk. Attempting to import as plugin.
[10/09/2023-15:59:37] [TRT] [I] Searching for plugin: GatherTopk, plugin_version: 1, plugin_namespace: 
[10/09/2023-15:59:37] [TRT] [I] Successfully created plugin: GatherTopk
[10/09/2023-15:59:37] [TRT] [I] No importer registered for op: GatherTopk. Attempting to import as plugin.
[10/09/2023-15:59:37] [TRT] [I] Searching for plugin: GatherTopk, plugin_version: 1, plugin_namespace: 
[10/09/2023-15:59:37] [TRT] [I] Successfully created plugin: GatherTopk
[10/09/2023-15:59:38] [TRT] [I] No importer registered for op: GatherTopk. Attempting to import as plugin.
[10/09/2023-15:59:38] [TRT] [I] Searching for plugin: GatherTopk, plugin_version: 1, plugin_namespace: 
[10/09/2023-15:59:38] [TRT] [I] Successfully created plugin: GatherTopk
[10/09/2023-15:59:38] [TRT] [I] No importer registered for op: GatherTopk. Attempting to import as plugin.
[10/09/2023-15:59:38] [TRT] [I] Searching for plugin: GatherTopk, plugin_version: 1, plugin_namespace: 
[10/09/2023-15:59:38] [TRT] [I] Successfully created plugin: GatherTopk
[10/09/2023-15:59:38] [TRT] [I] No importer registered for op: GatherTopk. Attempting to import as plugin.
[10/09/2023-15:59:38] [TRT] [I] Searching for plugin: GatherTopk, plugin_version: 1, plugin_namespace: 
[10/09/2023-15:59:38] [TRT] [I] Successfully created plugin: GatherTopk
[10/09/2023-15:59:38] [TRT] [I] No importer registered for op: GatherTopk. Attempting to import as plugin.
[10/09/2023-15:59:38] [TRT] [I] Searching for plugin: GatherTopk, plugin_version: 1, plugin_namespace: 
[10/09/2023-15:59:38] [TRT] [I] Successfully created plugin: GatherTopk
[10/09/2023-15:59:38] [TRT] [I] No importer registered for op: GatherTopk. Attempting to import as plugin.
[10/09/2023-15:59:38] [TRT] [I] Searching for plugin: GatherTopk, plugin_version: 1, plugin_namespace: 
[10/09/2023-15:59:38] [TRT] [I] Successfully created plugin: GatherTopk
[10/09/2023-15:59:38] [TRT] [I] No importer registered for op: GatherTopk. Attempting to import as plugin.
[10/09/2023-15:59:38] [TRT] [I] Searching for plugin: GatherTopk, plugin_version: 1, plugin_namespace: 
[10/09/2023-15:59:38] [TRT] [I] Successfully created plugin: GatherTopk
[10/09/2023-15:59:38] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[10/09/2023-15:59:39] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[10/09/2023-15:59:39] [TRT] [I] No importer registered for op: TRTBatchedNMS. Attempting to import as plugin.
[10/09/2023-15:59:39] [TRT] [I] Searching for plugin: TRTBatchedNMS, plugin_version: 1, plugin_namespace: 
[10/09/2023-15:59:39] [TRT] [I] Successfully created plugin: TRTBatchedNMS
[10/09/2023-15:59:39] [TRT] [I] No importer registered for op: MMCVMultiLevelRoiAlign. Attempting to import as plugin.
[10/09/2023-15:59:39] [TRT] [I] Searching for plugin: MMCVMultiLevelRoiAlign, plugin_version: 1, plugin_namespace: 
[10/09/2023-15:59:39] [TRT] [I] Successfully created plugin: MMCVMultiLevelRoiAlign
[10/09/2023-15:59:39] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[10/09/2023-15:59:39] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[10/09/2023-15:59:39] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[10/09/2023-15:59:40] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[10/09/2023-15:59:40] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[10/09/2023-15:59:40] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[10/09/2023-15:59:40] [TRT] [W] Tensor DataType is determined at build time for tensors not marked as input or output.
[10/09/2023-15:59:41] [TRT] [I] No importer registered for op: TRTBatchedNMS. Attempting to import as plugin.
[10/09/2023-15:59:41] [TRT] [I] Searching for plugin: TRTBatchedNMS, plugin_version: 1, plugin_namespace: 
[10/09/2023-15:59:41] [TRT] [I] Successfully created plugin: TRTBatchedNMS
[10/09/2023-15:59:41] [TRT] [W] Output type must be INT32 for shape outputs
[10/09/2023-15:59:43] [TRT] [E] 1: [ltWrapper.cpp::CublasLtWrapper::66] Error Code 1: Cublas (Could not initialize cublasLt. Please check CUDA installation.)
[10/09/2023-15:59:43] [TRT] [E] 2: [builder.cpp::buildSerializedNetwork::609] Error Code 2: Internal Error (Assertion enginePtr != nullptr failed. )
Process Process-3:
Traceback (most recent call last):
  File "/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py", line 107, in __call__
    ret = func(*args, **kwargs)
  File "/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/apis/utils/utils.py", line 98, in to_backend
    return backend_mgr.to_backend(
  File "/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/backend/tensorrt/backend_manager.py", line 127, in to_backend
    onnx2tensorrt(
  File "/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/backend/tensorrt/onnx2tensorrt.py", line 79, in onnx2tensorrt
    from_onnx(
  File "/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/backend/tensorrt/utils.py", line 248, in from_onnx
    assert engine is not None, 'Failed to create TensorRT engine'
AssertionError: Failed to create TensorRT engine
10/09 15:59:44 - mmengine - ERROR - /home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py - pop_mp_output - 80 - `mmdeploy.apis.utils.utils.to_backend` with Call id: 1 failed. exit.

Environment

10/09 15:52:30 - mmengine - INFO - 

10/09 15:52:30 - mmengine - INFO - **********Environmental information**********
10/09 15:52:31 - mmengine - INFO - sys.platform: linux
10/09 15:52:31 - mmengine - INFO - Python: 3.8.18 (default, Sep 11 2023, 13:40:15) [GCC 11.2.0]
10/09 15:52:31 - mmengine - INFO - CUDA available: True
10/09 15:52:31 - mmengine - INFO - numpy_random_seed: 2147483648
10/09 15:52:31 - mmengine - INFO - GPU 0,1,2,3,4,5,6,7: NVIDIA GeForce RTX 2080 Ti
10/09 15:52:31 - mmengine - INFO - CUDA_HOME: /usr/local/cuda-11.2
10/09 15:52:31 - mmengine - INFO - NVCC: Cuda compilation tools, release 11.2, V11.2.67
10/09 15:52:31 - mmengine - INFO - GCC: gcc (GCC) 6.5.1 20181026
10/09 15:52:31 - mmengine - INFO - PyTorch: 1.12.1+cu102
10/09 15:52:31 - mmengine - INFO - PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 10.2
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70
  - CuDNN 7.6.5
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.2, CUDNN_VERSION=7.6.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 

10/09 15:52:31 - mmengine - INFO - TorchVision: 0.13.1+cu102
10/09 15:52:31 - mmengine - INFO - OpenCV: 4.8.1
10/09 15:52:31 - mmengine - INFO - MMEngine: 0.8.5
10/09 15:52:31 - mmengine - INFO - MMCV: 2.0.1
10/09 15:52:31 - mmengine - INFO - MMCV Compiler: GCC 7.3
10/09 15:52:31 - mmengine - INFO - MMCV CUDA Compiler: 10.2
10/09 15:52:31 - mmengine - INFO - MMDeploy: 1.3.0+4c376d9
10/09 15:52:31 - mmengine - INFO - 

10/09 15:52:31 - mmengine - INFO - **********Backend information**********
10/09 15:52:31 - mmengine - INFO - tensorrt:	8.2.4.2
10/09 15:52:31 - mmengine - INFO - tensorrt custom ops:	Available
10/09 15:52:31 - mmengine - INFO - ONNXRuntime:	None
10/09 15:52:31 - mmengine - INFO - ONNXRuntime-gpu:	1.8.1
10/09 15:52:31 - mmengine - INFO - ONNXRuntime custom ops:	Available
10/09 15:52:31 - mmengine - INFO - pplnn:	None
10/09 15:52:31 - mmengine - INFO - ncnn:	None
10/09 15:52:31 - mmengine - INFO - snpe:	None
10/09 15:52:31 - mmengine - INFO - openvino:	None
10/09 15:52:31 - mmengine - INFO - torchscript:	1.12.1+cu102
10/09 15:52:31 - mmengine - INFO - torchscript custom ops:	NotAvailable
10/09 15:52:31 - mmengine - INFO - rknn-toolkit:	None
10/09 15:52:31 - mmengine - INFO - rknn-toolkit2:	None
10/09 15:52:31 - mmengine - INFO - ascend:	None
10/09 15:52:31 - mmengine - INFO - coreml:	None
10/09 15:52:31 - mmengine - INFO - tvm:	None
10/09 15:52:31 - mmengine - INFO - vacc:	None
10/09 15:52:31 - mmengine - INFO - 

10/09 15:52:31 - mmengine - INFO - **********Codebase information**********
10/09 15:52:31 - mmengine - INFO - mmdet:	3.1.0
10/09 15:52:31 - mmengine - INFO - mmseg:	None
10/09 15:52:31 - mmengine - INFO - mmpretrain:	None
10/09 15:52:31 - mmengine - INFO - mmocr:	None
10/09 15:52:31 - mmengine - INFO - mmagic:	None
10/09 15:52:31 - mmengine - INFO - mmdet3d:	None
10/09 15:52:31 - mmengine - INFO - mmpose:	None
10/09 15:52:31 - mmengine - INFO - mmrotate:	None
10/09 15:52:31 - mmengine - INFO - mmaction:	None
10/09 15:52:31 - mmengine - INFO - mmrazor:	None
10/09 15:52:31 - mmengine - INFO - mmyolo:	None

Error traceback

Traceback (most recent call last):
  File "/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py", line 107, in __call__
    ret = func(*args, **kwargs)
  File "/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/apis/utils/utils.py", line 98, in to_backend
    return backend_mgr.to_backend(
  File "/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/backend/tensorrt/backend_manager.py", line 127, in to_backend
    onnx2tensorrt(
  File "/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/backend/tensorrt/onnx2tensorrt.py", line 79, in onnx2tensorrt
    from_onnx(
  File "/home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/backend/tensorrt/utils.py", line 248, in from_onnx
    assert engine is not None, 'Failed to create TensorRT engine'
AssertionError: Failed to create TensorRT engine
10/09 15:42:57 - mmengine - ERROR - /home/admin/miniconda3/envs/mmdeploy/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py - pop_mp_output - 80 - `mmdeploy.apis.utils.utils.to_backend` with Call id: 1
 failed. exit.

TouchDeeper avatar Oct 09 '23 08:10 TouchDeeper

PyTorch: 1.12.1+cu102

可以试一下用cuda11.x版本的pytorch。

或者按照这个https://github.com/NVIDIA/TensorRT/issues/866#issuecomment-719758814 使用命令的形式来转tensorrt trtexec --onnx=xxx.onnx --tacticSources=-cublasLt,+cublas --workspace=2048 --fp16 --saveEngine=xxx.engine 用netron打开onnx模型看一下，可能需要设置input shape

irexyc avatar Oct 12 '23 02:10 irexyc

@irexyc 我尝试分别使用torch2onnx.py和onnx2tensorrt.py转换，转成功了

TouchDeeper avatar Oct 16 '23 08:10 TouchDeeper

@TouchDeeper 能否展开说一下如何分别使用torch2onnx.py和onnx2tensorrt.py转换，我遇到了同样的问题，期待您的回复！

Guangpengwgp avatar Dec 17 '23 15:12 Guangpengwgp

I have met same question, in another issue sb. mentioned that it was caused by insufficient CUDA memory. If anybody soved this questions, plz tell me :）

daohaofunk avatar Feb 04 '24 09:02 daohaofunk