RuntimeError: CUDA error: an illegal memory access was encountered when evaluate the beyond bounding-box ?

[Anm-pinellia]

Describe the bug I am trying to train and test the clf model which using the configs of rotated_reppoints_r50_fpn_1x_dota_oc, it is successed to finished the train process, but an error occured during the evaluation, here the log is:

File "E:/Experiment/目标检测实验/MMDet/OBB_Detectors/test.py", line 87, in predict outputs = single_gpu_test(model, data_loader, args.show, args.show_dir, args.show_score_thr) File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmdet\apis\test.py", line 31, in single_gpu_test result = model(return_loss=False, rescale=True, **data) File "E:\Anaconda\envs\mmdet2\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmcv\parallel\data_parallel.py", line 50, in forward return super().forward(*inputs, **kwargs) File "E:\Anaconda\envs\mmdet2\lib\site-packages\torch\nn\parallel\data_parallel.py", line 165, in forward return self.module(*inputs[0], **kwargs[0]) File "E:\Anaconda\envs\mmdet2\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmcv\runner\fp16_utils.py", line 109, in new_func return old_func(*args, **kwargs) File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmdet\models\detectors\base.py", line 174, in forward return self.forward_test(img, img_metas, **kwargs) File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmdet\models\detectors\base.py", line 147, in forward_test return self.simple_test(imgs[0], img_metas[0], **kwargs) File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmrotate\models\detectors\single_stage.py", line 100, in simple_test bbox_list = self.bbox_head.get_bboxes(*outs, img_metas, rescale=rescale) File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmcv\runner\fp16_utils.py", line 197, in new_func return old_func(*args, **kwargs) File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmrotate\models\dense_heads\rotated_reppoints_head.py", line 1044, in get_bboxes results = self._get_bboxes_single(cls_score_list, point_pred_list, File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmrotate\models\dense_heads\rotated_reppoints_head.py", line 1136, in _get_bboxes_single mlvl_bboxes[..., :4] /= mlvl_bboxes[..., :4].new_tensor(scale_factor) RuntimeError: CUDA error: an illegal memory access was encountered

Details

I have tired to find reason for this error, and find this is caused a specific data. Specifically, a function that named min_area_polygons in rotated_reppoints_head.py cannot handle the variables pts in the second iteration of processing.

The num_class in the model config is changed to 16 in my test to fit the DOTA V1.5 The data that caused this error is uploaded to baidu cloud driver if you need to test it:
链接：https://pan.baidu.com/s/1CCnthEl-kzOIXfJnU-PxRQ?pwd=l13h 提取码：l13h

3. What dataset did you use? DOTA_V1.5 splited with 1024 size and 500 gaps

Environment Other images can be predict correctly and the results of mmrotate/utils/collect_env.py are shown here: sys.platform: win32 Python: 3.8.12 (default, Oct 12 2021, 03:01:40) [MSC v.1916 64 bit (AMD64)] CUDA available: True GPU 0: GeForce GTX 1660 Ti CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2 NVCC: Cuda compilation tools, release 10.2, V10.2.89 GCC: n/a PyTorch: 1.8.1 PyTorch compiling details: PyTorch built with:

C++ Version: 199711 MSVC 192829913 Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683) OpenMP 2019 CPU capability usage: AVX2 CUDA Runtime 10.2 NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37 CuDNN 7.6.5 Magma 2.5.4 Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.2, CUDNN_VERSION=7.6.5, CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -DUSE_PTHREADPOOL -openmp:experimental -DNDEBUG -DUSE_FBGEMM -DUSE_XNNPACK, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, TorchVision: 0.9.1 OpenCV: 4.5.5 MMCV: 1.4.5 MMCV Compiler: MSVC 192930136 MMCV CUDA Compiler: 10.2 MMRotate: 0.1.1+

[zytx121]

Please add CUDA_LAUNCH_BLOCKING=1 and paste the error log.

[Anm-pinellia]

Please add CUDA_LAUNCH_BLOCKING=1 and paste the error log.

the same error occurred in this setting. 1 load checkpoint from local path: E:\Experiment\目标检测实验\MMDet\OBB_Detectors\Beyond_BoundingBox\work_dir\latest.pth [ ] 0/1, elapsed: 0s, ETA:Traceback (most recent call last): File "E:/Experiment/目标检测实验/MMDet/OBB_Detectors/test.py", line 143, in

File "E:/Experiment/目标检测实验/MMDet/OBB_Detectors/test.py", line 87, in predict outputs = single_gpu_test(model, data_loader, args.show, args.show_dir, args.show_score_thr) File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmdet\apis\test.py", line 29, in single_gpu_test result = model(return_loss=False, rescale=True, **data) File "E:\Anaconda\envs\mmdet2\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmcv\parallel\data_parallel.py", line 50, in forward return super().forward(*inputs, **kwargs) File "E:\Anaconda\envs\mmdet2\lib\site-packages\torch\nn\parallel\data_parallel.py", line 165, in forward return self.module(*inputs[0], **kwargs[0]) File "E:\Anaconda\envs\mmdet2\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmcv\runner\fp16_utils.py", line 109, in new_func return old_func(*args, **kwargs) File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmdet\models\detectors\base.py", line 174, in forward return self.forward_test(img, img_metas, **kwargs) File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmdet\models\detectors\base.py", line 147, in forward_test return self.simple_test(imgs[0], img_metas[0], **kwargs) File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmrotate\models\detectors\single_stage.py", line 100, in simple_test bbox_list = self.bbox_head.get_bboxes(*outs, img_metas, rescale=rescale) File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmcv\runner\fp16_utils.py", line 197, in new_func return old_func(*args, **kwargs) File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmrotate\models\dense_heads\rotated_reppoints_head.py", line 1044, in get_bboxes results = self._get_bboxes_single(cls_score_list, point_pred_list, File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmrotate\models\dense_heads\rotated_reppoints_head.py", line 1125, in _get_bboxes_single poly_pred = self.points2rotrect(points_pred, y_first=True) File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmrotate\models\dense_heads\rotated_reppoints_head.py", line 211, in points2rotrect rotrect_pred = min_area_polygons(pts) File "E:\Anaconda\envs\mmdet2\lib\site-packages\mmcv\ops\min_area_polygons.py", line 17, in min_area_polygons ext_module.min_area_polygons(pointsets, polygons) RuntimeError: CUDA error: an illegal memory access was encountered

[zytx121]

This error is caused by incorrect input of min_area_polygons. Could you print the pts in rotrect_pred = min_area_polygons(pts) and check it format?

[Anm-pinellia]

This error is caused by incorrect input of min_area_polygons. Could you print the pts in rotrect_pred = min_area_polygons(pts) and check it format?

the first pts are shown here: tensor([[-1.0003, -0.4358, 0.6460, ..., 1.0339, -0.3023, -0.3141], [-0.3291, -1.7686, 0.5437, ..., 1.3618, -0.1748, -0.6429], [-0.1825, -0.4383, 1.5624, ..., 0.8692, 0.1711, -0.2527], ..., [ 0.0894, -0.0731, 1.9433, ..., 0.7824, 0.4239, -0.0639], [-1.6038, -0.4877, 0.4154, ..., 3.1096, -0.4383, -0.1936], [-1.1776, -0.4550, 0.5818, ..., 0.3581, -0.4686, -0.7048]], device='cuda:0')

and the second pts are shown here: tensor([[-1.2346, -3.3156, 1.1556, ..., 2.6584, -0.0107, -0.4711], [-0.7851, -0.7225, 1.2545, ..., 0.6979, -0.0919, -0.3629], [-1.1499, -2.4557, 1.1019, ..., 3.6952, -0.1184, -0.0699], ..., [-1.8137, -2.7932, 2.7604, ..., 4.1961, 0.4080, 0.2027], [-2.8470, -1.2553, 3.2741, ..., 3.0502, -0.2177, -0.1452], [-1.8971, -0.9886, 2.0489, ..., 1.6850, -0.1780, -0.2618]], device='cuda:0')

I tested their performance in the function of min_area_polygons and found that the second pts will caused this error.

[zytx121]

It looks like a bug in min_area_polygons, we will check it.

[qixiong-wang]

I met the same error when testing rotate reppoint model.

[Anm-pinellia]

I met the same error when testing rotate reppoint model.

it seems a bug。。。

[TuanTNG]

I met the same error when testing rotate reppoint model.

I also have same error

[19990101lrk]

I have the same problem, is there a solution?

[yangxue0827]

A successful solution: set smaller nms_pre

test_cfg=dict(
        nms_pre=1000,
        min_bbox_size=0,
        score_thr=0.05,
        nms=dict(iou_thr=0.4),
        max_per_img=2000))

[austinmw]

@yangxue0827 That change did not solve the issue for me as mentioned in https://github.com/open-mmlab/mmrotate/issues/405#issuecomment-1215944571

[GisRookie]

I also have the same problem after change nms_pre.

[jiayuan666]

same error as mine, I have tried on Tesla V100-PCIE 32GB and RTX3090 26G when I modified the nms_pre to 20, it worked, but same problem occurred when num_pre > 30 here is error log where i add CUDA_LAUNCH_BLOCKING=1

[                           ] 232/16540, 19.4 task/s, elapsed: 12s, ETA:   840sTraceback (most recent call last):
  File "/home/jiayuan666/PycharmProjects/mmrotate/tools/test.py", line 257, in <module>
    main()
  File "/home/jiayuan666/PycharmProjects/mmrotate/tools/test.py", line 222, in main
    outputs = single_gpu_test(model, data_loader, args.show, args.show_dir,
  File "/home/jiayuan666/.conda/envs/mmrotate0.3.3/lib/python3.8/site-packages/mmdet/apis/test.py", line 29, in single_gpu_test
    result = model(return_loss=False, rescale=True, **data)
  File "/home/jiayuan666/.conda/envs/mmrotate0.3.3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/jiayuan666/.conda/envs/mmrotate0.3.3/lib/python3.8/site-packages/mmcv/parallel/data_parallel.py", line 51, in forward
    return super().forward(*inputs, **kwargs)
  File "/home/jiayuan666/.conda/envs/mmrotate0.3.3/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 166, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/jiayuan666/.conda/envs/mmrotate0.3.3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/jiayuan666/.conda/envs/mmrotate0.3.3/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 119, in new_func
    return old_func(*args, **kwargs)
  File "/home/jiayuan666/.conda/envs/mmrotate0.3.3/lib/python3.8/site-packages/mmdet/models/detectors/base.py", line 174, in forward
    return self.forward_test(img, img_metas, **kwargs)
  File "/home/jiayuan666/.conda/envs/mmrotate0.3.3/lib/python3.8/site-packages/mmdet/models/detectors/base.py", line 147, in forward_test
    return self.simple_test(imgs[0], img_metas[0], **kwargs)
  File "/home/jiayuan666/PycharmProjects/mmrotate/mmrotate/models/detectors/single_stage.py", line 101, in simple_test
    bbox_list = self.bbox_head.get_bboxes(
  File "/home/jiayuan666/.conda/envs/mmrotate0.3.3/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 208, in new_func
    return old_func(*args, **kwargs)
  File "/home/jiayuan666/PycharmProjects/mmrotate/mmrotate/models/dense_heads/rotated_reppoints_head.py", line 1066, in get_bboxes
    results = self._get_bboxes_single(cls_score_list, point_pred_list,
  File "/home/jiayuan666/PycharmProjects/mmrotate/mmrotate/models/dense_heads/rotated_reppoints_head.py", line 1148, in _get_bboxes_single
    poly_pred = self.points2rotrect(points_pred, y_first=True)
  File "/home/jiayuan666/PycharmProjects/mmrotate/mmrotate/models/dense_heads/rotated_reppoints_head.py", line 211, in points2rotrect
    rotrect_pred = min_area_polygons(pts)
  File "/home/jiayuan666/.conda/envs/mmrotate0.3.3/lib/python3.8/site-packages/mmcv/ops/min_area_polygons.py", line 19, in min_area_polygons
    ext_module.min_area_polygons(pointsets, polygons)
RuntimeError: CUDA error: an illegal memory access was encountered
Exception raised from MinAreaPolygonsCUDAKernelLauncher at /tmp/mmcv/mmcv/ops/csrc/pytorch/cuda/min_area_polygons.cu:20 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f5dae0c5497 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10::CUDAError::Error(c10::SourceLocation, std::string) + 0x30 (0x7f5d6b111f84 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/lib/python3.8/site-packages/mmcv/_ext.cpython-38-x86_64-linux-gnu.so)
frame #2: MinAreaPolygonsCUDAKernelLauncher(at::Tensor, at::Tensor) + 0x17e (0x7f5d6b186b35 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/lib/python3.8/site-packages/mmcv/_ext.cpython-38-x86_64-linux-gnu.so)
frame #3: min_area_polygons_cuda(at::Tensor, at::Tensor) + 0x49 (0x7f5d6b14b9c9 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/lib/python3.8/site-packages/mmcv/_ext.cpython-38-x86_64-linux-gnu.so)
frame #4: auto Dispatch<DeviceRegistry<void (*)(at::Tensor, at::Tensor), &(min_area_polygons_impl(at::Tensor, at::Tensor))>, at::Tensor const&, at::Tensor&>(DeviceRegistry<void (*)(at::Tensor, at::Tensor), &(min_area_polygons_impl(at::Tensor, at::Tensor))> const&, char const*, at::Tensor const&, at::Tensor&) + 0xb7 (0x7f5d6b29b527 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/lib/python3.8/site-packages/mmcv/_ext.cpython-38-x86_64-linux-gnu.so)
frame #5: min_area_polygons(at::Tensor, at::Tensor) + 0x49 (0x7f5d6b29b3a9 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/lib/python3.8/site-packages/mmcv/_ext.cpython-38-x86_64-linux-gnu.so)
frame #6: <unknown function> + 0x2bb89d (0x7f5d6b2c089d in /home/jiayuan666/.conda/envs/mmrotate0.3.3/lib/python3.8/site-packages/mmcv/_ext.cpython-38-x86_64-linux-gnu.so)
frame #7: <unknown function> + 0x2d12db (0x7f5d6b2d62db in /home/jiayuan666/.conda/envs/mmrotate0.3.3/lib/python3.8/site-packages/mmcv/_ext.cpython-38-x86_64-linux-gnu.so)
frame #8: PyCFunction_Call + 0x52 (0x4dfd82 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #9: _PyObject_MakeTpCall + 0x3eb (0x4d0c5b in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #10: _PyEval_EvalFrameDefault + 0x5265 (0x4cc005 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #11: _PyFunction_Vectorcall + 0x106 (0x4d9d16 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #12: _PyEval_EvalFrameDefault + 0x907 (0x4c76a7 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #13: _PyEval_EvalCodeWithName + 0x1f5 (0x4c5c45 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #14: /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python() [0x4e8224]
frame #15: _PyEval_EvalFrameDefault + 0x172a (0x4c84ca in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #16: _PyEval_EvalCodeWithName + 0x1f5 (0x4c5c45 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #17: _PyFunction_Vectorcall + 0x19c (0x4d9dac in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #18: /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python() [0x4e8197]
frame #19: PyObject_Call + 0x5e (0x4ec16e in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #20: _PyEval_EvalFrameDefault + 0x2051 (0x4c8df1 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #21: _PyEval_EvalCodeWithName + 0x1f5 (0x4c5c45 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #22: _PyFunction_Vectorcall + 0x19c (0x4d9dac in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #23: PyObject_Call + 0x5e (0x4ec16e in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #24: _PyEval_EvalFrameDefault + 0x2051 (0x4c8df1 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #25: _PyEval_EvalCodeWithName + 0x1f5 (0x4c5c45 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #26: _PyFunction_Vectorcall + 0x19c (0x4d9dac in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #27: /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python() [0x4e8197]
frame #28: PyObject_Call + 0x5e (0x4ec16e in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #29: _PyEval_EvalFrameDefault + 0x2051 (0x4c8df1 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #30: _PyEval_EvalCodeWithName + 0x1f5 (0x4c5c45 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #31: _PyFunction_Vectorcall + 0x19c (0x4d9dac in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #32: /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python() [0x4e8197]
frame #33: PyObject_Call + 0x5e (0x4ec16e in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #34: _PyEval_EvalFrameDefault + 0x2051 (0x4c8df1 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #35: _PyEval_EvalCodeWithName + 0x1f5 (0x4c5c45 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #36: _PyFunction_Vectorcall + 0x19c (0x4d9dac in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #37: /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python() [0x4e8197]
frame #38: PyObject_Call + 0x5e (0x4ec16e in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #39: _PyEval_EvalFrameDefault + 0x2051 (0x4c8df1 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #40: _PyEval_EvalCodeWithName + 0x1f5 (0x4c5c45 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #41: _PyFunction_Vectorcall + 0x19c (0x4d9dac in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #42: PyObject_Call + 0x5e (0x4ec16e in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #43: _PyEval_EvalFrameDefault + 0x2051 (0x4c8df1 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #44: _PyEval_EvalCodeWithName + 0x1f5 (0x4c5c45 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #45: _PyFunction_Vectorcall + 0x19c (0x4d9dac in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #46: /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python() [0x4e8197]
frame #47: PyObject_Call + 0x5e (0x4ec16e in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #48: _PyEval_EvalFrameDefault + 0x2051 (0x4c8df1 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #49: _PyEval_EvalCodeWithName + 0x1f5 (0x4c5c45 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #50: _PyFunction_Vectorcall + 0x19c (0x4d9dac in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #51: _PyObject_FastCallDict + 0x25f (0x4d028f in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #52: _PyObject_Call_Prepend + 0x60 (0x4e4720 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #53: /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python() [0x541f07]
frame #54: PyObject_Call + 0x272 (0x4ec382 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #55: _PyEval_EvalFrameDefault + 0x2051 (0x4c8df1 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #56: _PyEval_EvalCodeWithName + 0x1f5 (0x4c5c45 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #57: _PyFunction_Vectorcall + 0x19c (0x4d9dac in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #58: /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python() [0x4e8197]
frame #59: PyObject_Call + 0x5e (0x4ec16e in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #60: _PyEval_EvalFrameDefault + 0x2051 (0x4c8df1 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #61: _PyEval_EvalCodeWithName + 0x1f5 (0x4c5c45 in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #62: _PyFunction_Vectorcall + 0x19c (0x4d9dac in /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python)
frame #63: /home/jiayuan666/.conda/envs/mmrotate0.3.3/bin/python() [0x4e8197]

[sunny-sjj]

I want to know if this bug is solved?

[pphgood]

I also have the same problem after change nms_pre. I want to know if this bug is solved?

[pphgood]

I want to know if this bug is solved?

A successful solution: set smaller nms_pre

test_cfg=dict(
        nms_pre=1000,
        min_bbox_size=0,
        score_thr=0.05,
        nms=dict(iou_thr=0.4),
        max_per_img=2000))

I also have the same problem after change nms_pre. I found this error when I was testing a certain image, but not other images. I want to know if this bug is solved?

[freshn]

Same here.

[ToneZe]

The same problem. sys.platform: linux Python: 3.8.13 (default, Apr 19 2022, 00:53:22) [GCC 7.5.0] CUDA available: True GPU 0: Tesla V100-PCIE-16GB CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 10.2, V10.2.8 GCC: x86_64-linux-gnu-gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 PyTorch: 1.8.1+cu102 PyTorch compiling details: PyTorch built with:

• GCC 7.3
• C++ Version: 201402
• Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• NNPACK is enabled
• CPU capability usage: AVX2
• CUDA Runtime 10.2
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70
• CuDNN 7.6.5
• Magma 2.5.2
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.2, CUDNN_VERSION=7.6.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.9.1+cu102 OpenCV: 4.8.0 MMCV: 1.6.1 MMCV Compiler: GCC 7.3 MMCV CUDA Compiler: 10.2 MMRotate: 0.3.4+

File "./tools/train.py", line 192, in if name == 'main': File "./tools/train.py", line 181, in main model.CLASSES = datasets[0].CLASSES File "/hy-tmp/LSKNet/mmrotate/apis/train.py", line 141, in train_detector runner.run(data_loaders, cfg.workflow) File "/usr/local/lib/python3.8/dist-packages/mmcv/runner/epoch_based_runner.py", line 136, in run epoch_runner(data_loaders[i], **kwargs) File "/usr/local/lib/python3.8/dist-packages/mmcv/runner/epoch_based_runner.py", line 58, in train self.call_hook('after_train_epoch') File "/usr/local/lib/python3.8/dist-packages/mmcv/runner/base_runner.py", line 317, in call_hook getattr(hook, fn_name)(self) File "/usr/local/lib/python3.8/dist-packages/mmcv/runner/hooks/evaluation.py", line 271, in after_train_epoch self._do_evaluate(runner) File "/usr/local/lib/python3.8/dist-packages/mmdet/core/evaluation/eval_hooks.py", line 126, in _do_evaluate results = multi_gpu_test( File "/usr/local/lib/python3.8/dist-packages/mmdet/apis/test.py", line 109, in multi_gpu_test result = model(return_loss=False, rescale=True, **data) File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py", line 705, in forward output = self.module(*inputs[0], **kwargs[0]) File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/usr/local/lib/python3.8/dist-packages/mmcv/runner/fp16_utils.py", line 116, in new_func return old_func(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/mmdet/models/detectors/base.py", line 174, in forward return self.forward_test(img, img_metas, **kwargs) File "/usr/local/lib/python3.8/dist-packages/mmdet/models/detectors/base.py", line 147, in forward_test return self.simple_test(imgs[0], img_metas[0], **kwargs) File "/hy-tmp/LSKNet/mmrotate/models/detectors/single_stage.py", line 101, in simple_test bbox_list = self.bbox_head.get_bboxes( File "/usr/local/lib/python3.8/dist-packages/mmcv/runner/fp16_utils.py", line 205, in new_func return old_func(args, kwargs) File "/hy-tmp/LSKNet/mmrotate/models/dense_heads/sam_reppoints_head.py", line 734, in get_bboxes results = self._get_bboxes_single(cls_score_list, point_pred_list, File "/hy-tmp/LSKNet/mmrotate/models/dense_heads/sam_reppoints_head.py", line 828, in _get_bboxes_single mlvl_bboxes[..., :4] /= mlvl_bboxes[..., :4].new_tensor( RuntimeError: CUDA error: an illegal memory access was encountered terminate called after throwing an instance of 'c10::Error' what(): CUDA error: an illegal memory access was encountered Exception raised from create_event_internal at /pytorch/c10/cuda/CUDACachingAllocator.cpp:733 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f76727062f2 in /usr/local/lib/python3.8/dist-packages/torch/lib/libc10.so) frame #1: c10::detail::torchCheckFail(char const, char const, unsigned int, std::string const&) + 0x5b (0x7f767270367b in /usr/local/lib/python3.8/dist-packages/torch/lib/libc10.so) frame #2: c10::cuda::CUDACachingAllocator::raw_delete(void) + 0x809 (0x7f767295e1f9 in /usr/local/lib/python3.8/dist-packages/torch/lib/libc10_cuda.so) frame #3: c10::TensorImpl::release_resources() + 0x54 (0x7f76726ee3a4 in /usr/local/lib/python3.8/dist-packages/torch/lib/libc10.so) frame #4: + 0x6e43ca (0x7f76226593ca in /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_python.so) frame #5: + 0x6e4461 (0x7f7622659461 in /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_python.so) frame #6: /usr/bin/python() [0x53f4c3] frame #7: /usr/bin/python() [0x58d35c] frame #8: /usr/bin/python() [0x57945b] frame #9: /usr/bin/python() [0x53de5c] frame #10: /usr/bin/python() [0x5abb4c] frame #11: /usr/bin/python() [0x5bcee8] frame #12: /usr/bin/python() [0x5bce4e] frame #13: /usr/bin/python() [0x5bce4e] frame #14: /usr/bin/python() [0x5bce4e] frame #15: /usr/bin/python() [0x5bce4e] frame #16: /usr/bin/python() [0x5bce4e] frame #17: /usr/bin/python() [0x5bce4e] frame #18: /usr/bin/python() [0x5bce4e] frame #19: /usr/bin/python() [0x5bce4e] frame #20: /usr/bin/python() [0x5bce4e] frame #21: /usr/bin/python() [0x5bce4e] frame #22: /usr/bin/python() [0x56ed56] frame #23: PyDict_SetItemString + 0x50 (0x5751a0 in /usr/bin/python) frame #24: PyImport_Cleanup + 0x76 (0x64e5a6 in /usr/bin/python) frame #25: Py_FinalizeEx + 0x6e (0x6407ce in /usr/bin/python) frame #26: Py_RunMain + 0xf9 (0x671c79 in /usr/bin/python) frame #27: Py_BytesMain + 0x29 (0x672009 in /usr/bin/python) frame #28: __libc_start_main + 0xe7 (0x7f7684b63bf7 in /lib/x86_64-linux-gnu/libc.so.6) frame #29: _start + 0x2a (0x5e201a in /usr/bin/python)

When training in epoch=50, this error is uncomfortable Killing subprocess 506 Traceback (most recent call last): File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.8/dist-packages/torch/distributed/launch.py", line 340, in main() File "/usr/local/lib/python3.8/dist-packages/torch/distributed/launch.py", line 326, in main sigkill_handler(signal.SIGTERM, None) # not coming back File "/usr/local/lib/python3.8/dist-packages/torch/distributed/launch.py", line 301, in sigkill_handler raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)

[sgtojd]

I wonder that have the this bug been solved?

[walkerinrain]

I find this bug result from some bad labels. After image split to small size, some bounding box will be cut apart，then some extremely narrow bounding boxes will be generated nearby the new image border. After forward propagation through the network, these bounding boxes will cause the network to generate prediction tensors with particularly large dimensions, then the training process was terminated due to exceeding the computational capacity. once we know the reason, then the solution is easy to find. Below are two solutions: 1.we can find the image and label being processed once training process was terminated, then delete them; 2.find bad labels like below and delete them

[Anm-pinellia]

I find this bug result from some bad labels. After image split to small size, some bounding box will be cut apart，then some extremely narrow bounding boxes will be generated nearby the new image border. After forward propagation through the network, these bounding boxes will cause the network to generate prediction tensors with particularly large dimensions, then the training process was terminated due to exceeding the computational capacity. once we know the reason, then the solution is easy to find. Below are two solutions: 1.we can find the image and label being processed once training process was terminated, then delete them; 2.find bad labels like below and delete them

Thanks for your kind suggestion. This is a good way to solve this problem.