mmdeploy #assertionD:\MMbushu\mmdeploy\csrc\mmdeploy\backend_ops\tensorrt\batched_nms\trt_batched_nms.cpp,103 2022-07-18 11:34:14,224 - mmdeploy - ERROR

Thanks for your bug report. We appreciate it a lot.

Checklist

I have searched related issues but cannot get the expected help.
I have read the FAQ documentation but cannot get the expected help.
The bug has not been fixed in the latest version.

Describe the bug #assertionD:\MMbushu\mmdeploy\csrc\mmdeploy\backend_ops\tensorrt\batched_nms\trt_batched_nms.cpp,103 2022-07-18 11:34:14,224 - mmdeploy - ERROR - visualize tensorrt model failed.

Reproduction

What command or script did you run?

python deploy.py --deploy_cfg D:\MMbushu\mmdeploy\configs\mmrotate\rotated-detection_tensorrt_dynamic-320x320-1024x1024.py --model_cfg oriented_rcnn_r50_fpn_1x_dota_le90.py --checkpoint latest.pth --img a_2022-05-19-16-13-02.jpg --work-dir work_dir/5 --show --log-level INFO --dump-info --device cuda:0 A placeholder for the command.


2. Did you make any modifications on the code or config? Did you understand what you have modified?

**Environment**

1. Please run `python tools/check_env.py` to collect necessary environment information and paste it here.
2. You may add addition that may be helpful for locating the problem, such as
   - How you installed PyTorch \[e.g., pip, conda, source\]
   - Other environment variables that may be related (such as `$PATH`, `$LD_LIBRARY_PATH`, `$PYTHONPATH`, etc.)

**Error traceback**

2022-07-18 11:26:58,529 - mmdeploy - INFO - Start pipeline mmdeploy.apis.pytorch2onnx.torch2onnx in subprocess
D:\anaconda3\envs\mmdeploy\lib\site-packages\mmdet\models\dense_heads\anchor_head.py:116: UserWarning: DeprecationWarning: `num_anchors` is deprecated, for consistency or also use `num_base_priors` instead
  warnings.warn('DeprecationWarning: `num_anchors` is deprecated, '
load checkpoint from local path: latest.pth
2022-07-18 11:27:02,967 - mmdeploy - WARNING - DeprecationWarning: get_onnx_config will be deprecated in the future.
2022-07-18 11:27:02,968 - mmdeploy - INFO - Export PyTorch model to ONNX: work_dir/5\end2end.onnx.
d:\mmbushu\mmdeploy\mmdeploy\core\optimizers\function_marker.py:158: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  ys_shape = tuple(int(s) for s in ys.shape)
D:\anaconda3\envs\mmdeploy\lib\site-packages\mmdet\models\dense_heads\anchor_head.py:123: UserWarning: DeprecationWarning: anchor_generator is deprecated, please use "prior_generator" instead
  warnings.warn('DeprecationWarning: anchor_generator is deprecated, '
D:\anaconda3\envs\mmdeploy\lib\site-packages\mmdet\core\anchor\anchor_generator.py:333: UserWarning: ``grid_anchors`` would be deprecated soon. Please use ``grid_priors``
  warnings.warn('``grid_anchors`` would be deprecated soon. '
D:\anaconda3\envs\mmdeploy\lib\site-packages\mmdet\core\anchor\anchor_generator.py:369: UserWarning: ``single_level_grid_anchors`` would be deprecated soon. Please use ``single_level_grid_priors``
  warnings.warn(
d:\mmbushu\mmdeploy\mmdeploy\codebase\mmrotate\models\rotated_rpn_head.py:79: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert cls_score.size()[-2:] == bbox_pred.size()[-2:]
d:\mmbushu\mmdeploy\mmdeploy\pytorch\functions\topk.py:56: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if k > size:
D:\anaconda3\envs\mmdeploy\lib\site-packages\mmrotate\core\bbox\coder\delta_midpointoffset_rbbox_coder.py:78: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert pred_bboxes.size(0) == bboxes.size(0)
D:\anaconda3\envs\mmdeploy\lib\site-packages\mmrotate\core\bbox\coder\delta_midpointoffset_rbbox_coder.py:79: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert bboxes.size(-1) == 4
D:\anaconda3\envs\mmdeploy\lib\site-packages\mmrotate\core\bbox\coder\delta_midpointoffset_rbbox_coder.py:80: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert pred_bboxes.size(-1) == 6
d:\mmbushu\mmdeploy\mmdeploy\codebase\mmrotate\core\post_processing\bbox_nms.py:276: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  int(scores.shape[-1]),
d:\mmbushu\mmdeploy\mmdeploy\mmcv\ops\nms.py:178: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  out_boxes = min(num_boxes, after_topk)
D:\anaconda3\envs\mmdeploy\lib\site-packages\mmrotate\core\bbox\coder\delta_xywha_rbbox_coder.py:101: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert pred_bboxes.size(0) == bboxes.size(0)
d:\mmbushu\mmdeploy\mmdeploy\codebase\mmrotate\core\post_processing\bbox_nms.py:163: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  int(scores.shape[-1]),
d:\mmbushu\mmdeploy\mmdeploy\mmcv\ops\nms_rotated.py:120: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  out_boxes = min(num_boxes, after_topk)
D:\anaconda3\envs\mmdeploy\lib\site-packages\torch\onnx\symbolic_opset9.py:2603: UserWarning: Exporting aten::index operator of advanced indexing in opset 11 is achieved by combination of multiple ONNX operators, including Reshape, Transpose, Concat, and Gather. If indices include negative values, the exported graph will produce incorrect results.
  warnings.warn("Exporting aten::index operator of advanced indexing in opset " +
2022-07-18 11:27:12,270 - mmdeploy - INFO - Execute onnx optimize passes.
2022-07-18 11:27:12,270 - mmdeploy - WARNING - Can not optimize model, please build torchscipt extension.
More details: https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/experimental/onnx_optimizer.md
2022-07-18 11:27:12,817 - mmdeploy - INFO - Finish pipeline mmdeploy.apis.pytorch2onnx.torch2onnx
2022-07-18 11:27:15,372 - mmdeploy - INFO - Start pipeline mmdeploy.backend.tensorrt.onnx2tensorrt.onnx2tensorrt in subprocess
2022-07-18 11:27:15,571 - mmdeploy - INFO - Successfully loaded tensorrt plugins from d:\mmbushu\mmdeploy\mmdeploy\lib\mmdeploy_tensorrt_ops.dll
[07/18/2022-11:27:15] [TRT] [I] [MemUsageChange] Init CUDA: CPU +512, GPU +0, now: CPU 7531, GPU 1207 (MiB)
[07/18/2022-11:27:16] [TRT] [I] [MemUsageSnapshot] Begin constructing builder kernel library: CPU 7586 MiB, GPU 1207 MiB
[07/18/2022-11:27:16] [TRT] [I] [MemUsageSnapshot] End constructing builder kernel library: CPU 7750 MiB, GPU 1251 MiB
[07/18/2022-11:27:16] [TRT] [W] onnx2trt_utils.cpp:366: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[07/18/2022-11:27:16] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[07/18/2022-11:27:19] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[07/18/2022-11:27:19] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[07/18/2022-11:27:19] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[07/18/2022-11:27:19] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[07/18/2022-11:27:19] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[07/18/2022-11:27:19] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[07/18/2022-11:27:20] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[07/18/2022-11:27:20] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[07/18/2022-11:27:20] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[07/18/2022-11:27:20] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[07/18/2022-11:27:21] [TRT] [I] No importer registered for op: TRTBatchedNMS. Attempting to import as plugin.
[07/18/2022-11:27:21] [TRT] [I] Searching for plugin: TRTBatchedNMS, plugin_version: 1, plugin_namespace:
[07/18/2022-11:27:21] [TRT] [I] Successfully created plugin: TRTBatchedNMS
[07/18/2022-11:27:22] [TRT] [I] No importer registered for op: MMCVMultiLevelRotatedRoiAlign. Attempting to import as plugin.
[07/18/2022-11:27:22] [TRT] [I] Searching for plugin: MMCVMultiLevelRotatedRoiAlign, plugin_version: 1, plugin_namespace:
[07/18/2022-11:27:22] [TRT] [I] Successfully created plugin: MMCVMultiLevelRotatedRoiAlign
[07/18/2022-11:27:22] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[07/18/2022-11:27:22] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[07/18/2022-11:27:22] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[07/18/2022-11:27:22] [TRT] [W] onnx2trt_utils.cpp:392: One or more weights outside the range of INT32 was clamped
[07/18/2022-11:27:24] [TRT] [I] No importer registered for op: TRTBatchedRotatedNMS. Attempting to import as plugin.
[07/18/2022-11:27:24] [TRT] [I] Searching for plugin: TRTBatchedRotatedNMS, plugin_version: 1, plugin_namespace:
[07/18/2022-11:27:24] [TRT] [I] Successfully created plugin: TRTBatchedRotatedNMS
[07/18/2022-11:27:24] [TRT] [W] Output type must be INT32 for shape outputs
[07/18/2022-11:27:24] [TRT] [W] TensorRT was linked against cuBLAS/cuBLAS LT 11.6.3 but loaded cuBLAS/cuBLAS LT 11.3.0
[07/18/2022-11:27:24] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +730, GPU +266, now: CPU 8744, GPU 1517 (MiB)
[07/18/2022-11:27:25] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +463, GPU +258, now: CPU 9207, GPU 1775 (MiB)
[07/18/2022-11:27:25] [TRT] [W] TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.0.5
[07/18/2022-11:27:25] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[07/18/2022-11:28:05] [TRT] [I] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[07/18/2022-11:30:08] [TRT] [W] Myelin graph with multiple dynamic values may have poor performance if they differ. Dynamic values are:
[07/18/2022-11:30:08] [TRT] [W]  (# 2 (SHAPE input))
[07/18/2022-11:30:08] [TRT] [W]  (# 3 (SHAPE input))
[07/18/2022-11:30:44] [TRT] [W] Myelin graph with multiple dynamic values may have poor performance if they differ. Dynamic values are:
[07/18/2022-11:30:44] [TRT] [W]  (# 2 (SHAPE input))
[07/18/2022-11:30:44] [TRT] [W]  (# 3 (SHAPE input))
[07/18/2022-11:30:45] [TRT] [W] Myelin graph with multiple dynamic values may have poor performance if they differ. Dynamic values are:
[07/18/2022-11:30:45] [TRT] [W]  (# 2 (SHAPE input))
[07/18/2022-11:30:45] [TRT] [W]  (# 3 (SHAPE input))
[07/18/2022-11:33:28] [TRT] [I] Detected 1 inputs and 2 output network tensors.
[07/18/2022-11:33:28] [TRT] [W] Myelin graph with multiple dynamic values may have poor performance if they differ. Dynamic values are:
[07/18/2022-11:33:28] [TRT] [W]  (# 2 (SHAPE input))
[07/18/2022-11:33:28] [TRT] [W]  (# 3 (SHAPE input))
[07/18/2022-11:33:28] [TRT] [W] Myelin graph with multiple dynamic values may have poor performance if they differ. Dynamic values are:
[07/18/2022-11:33:28] [TRT] [W]  (# 2 (SHAPE input))
[07/18/2022-11:33:28] [TRT] [W]  (# 3 (SHAPE input))
[07/18/2022-11:33:28] [TRT] [W] Myelin graph with multiple dynamic values may have poor performance if they differ. Dynamic values are:
[07/18/2022-11:33:28] [TRT] [W]  (# 2 (SHAPE input))
[07/18/2022-11:33:28] [TRT] [W]  (# 3 (SHAPE input))
[07/18/2022-11:33:28] [TRT] [I] Total Host Persistent Memory: 153984
[07/18/2022-11:33:28] [TRT] [I] Total Device Persistent Memory: 97400832
[07/18/2022-11:33:28] [TRT] [I] Total Scratch Memory: 19923456
[07/18/2022-11:33:28] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 9 MiB, GPU 745 MiB
[07/18/2022-11:33:29] [TRT] [I] [BlockAssignment] Algorithm ShiftNTopDown took 1112.34ms to assign 58 blocks to 375 nodes requiring 279896064 bytes.
[07/18/2022-11:33:29] [TRT] [I] Total Activation Memory: 279896064
[07/18/2022-11:33:29] [TRT] [W] TensorRT was linked against cuBLAS/cuBLAS LT 11.6.3 but loaded cuBLAS/cuBLAS LT 11.3.0
[07/18/2022-11:33:29] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 9889, GPU 2181 (MiB)
[07/18/2022-11:33:29] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 9889, GPU 2189 (MiB)
[07/18/2022-11:33:29] [TRT] [W] TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.0.5
[07/18/2022-11:33:29] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in building engine: CPU +0, GPU +167, now: CPU 0, GPU 167 (MiB)
2022-07-18 11:33:30,059 - mmdeploy - INFO - Finish pipeline mmdeploy.backend.tensorrt.onnx2tensorrt.onnx2tensorrt
2022-07-18 11:33:31,021 - mmdeploy - INFO - visualize tensorrt model start.
2022-07-18 11:33:35,170 - mmdeploy - INFO - Successfully loaded tensorrt plugins from d:\mmbushu\mmdeploy\mmdeploy\lib\mmdeploy_tensorrt_ops.dll
2022-07-18:11:33:35 - mmdeploy - INFO - Successfully loaded tensorrt plugins from d:\mmbushu\mmdeploy\mmdeploy\lib\mmdeploy_tensorrt_ops.dll
2022-07-18 11:33:35,171 - mmdeploy - INFO - Successfully loaded tensorrt plugins from d:\mmbushu\mmdeploy\mmdeploy\lib\mmdeploy_tensorrt_ops.dll
2022-07-18:11:33:35 - mmdeploy - INFO - Successfully loaded tensorrt plugins from d:\mmbushu\mmdeploy\mmdeploy\lib\mmdeploy_tensorrt_ops.dll
[07/18/2022-11:33:36] [TRT] [W] TensorRT was linked against cuBLAS/cuBLAS LT 11.6.3 but loaded cuBLAS/cuBLAS LT 11.3.0
[07/18/2022-11:33:36] [TRT] [W] TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.0.5
[07/18/2022-11:33:36] [TRT] [W] TensorRT was linked against cuBLAS/cuBLAS LT 11.6.3 but loaded cuBLAS/cuBLAS LT 11.3.0
[07/18/2022-11:33:36] [TRT] [W] TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.0.5
#assertionD:\MMbushu\mmdeploy\csrc\mmdeploy\backend_ops\tensorrt\batched_nms\trt_batched_nms.cpp,103
2022-07-18 11:34:14,224 - mmdeploy - ERROR - visualize tensorrt model failed.

```none
A placeholder for trackback.

Bug fix

If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!