ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE

[Vipermdl]

Hi, I`m facing the problem with training: (py36) [madongliang@compute-0-5 experiments]$ sh train.sh Traceback (most recent call last): Traceback (most recent call last): File "../mmdetection/tools/train.py", line 8, in File "../mmdetection/tools/train.py", line 8, in Traceback (most recent call last): File "../mmdetection/tools/train.py", line 8, in Traceback (most recent call last): File "../mmdetection/tools/train.py", line 8, in Traceback (most recent call last): Traceback (most recent call last): File "../mmdetection/tools/train.py", line 8, in Traceback (most recent call last): from mmdet.apis import (train_detector, init_dist, get_root_logger, from mmdet.apis import (train_detector, init_dist, get_root_logger, File "../mmdetection/tools/train.py", line 8, in File "../mmdetection/tools/train.py", line 8, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in from mmdet.apis import (train_detector, init_dist, get_root_logger, File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in from mmdet.apis import (train_detector, init_dist, get_root_logger, File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in Traceback (most recent call last): File "../mmdetection/tools/train.py", line 8, in from mmdet.apis import (train_detector, init_dist, get_root_logger, from mmdet.apis import (train_detector, init_dist, get_root_logger, File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in from mmdet.apis import (train_detector, init_dist, get_root_logger, File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in from mmdet.apis import (train_detector, init_dist, get_root_logger, File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/init.py", line 2, in from .train import train_detector File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in from .train import train_detector File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in from .train import train_detector File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in from .train import train_detector File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in from .train import train_detector File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in from .train import train_detector File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in from .train import train_detector File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in from mmdet.core import (DistOptimizerHook, DistEvalmAPHook, File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in from mmdet.core import (DistOptimizerHook, DistEvalmAPHook, from .train import train_detector File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/apis/train.py", line 9, in from mmdet.core import (DistOptimizerHook, DistEvalmAPHook, from mmdet.core import (DistOptimizerHook, DistEvalmAPHook, from mmdet.core import (DistOptimizerHook, DistEvalmAPHook, File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in from mmdet.core import (DistOptimizerHook, DistEvalmAPHook, from mmdet.core import (DistOptimizerHook, DistEvalmAPHook, File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in from .post_processing import * # noqa: F401, F403 File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in from mmdet.core import (DistOptimizerHook, DistEvalmAPHook, from .post_processing import * # noqa: F401, F403 File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in from .post_processing import * # noqa: F401, F403 File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/init.py", line 6, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in from .post_processing import * # noqa: F401, F403 File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in from .post_processing import * # noqa: F401, F403 from .post_processing import * # noqa: F401, F403 File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in from .post_processing import * # noqa: F401, F403 File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in from .post_processing import * # noqa: F401, F403 File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/init.py", line 1, in from .bbox_nms import multiclass_nms File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in from .bbox_nms import multiclass_nms File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in from .bbox_nms import multiclass_nms from .bbox_nms import multiclass_nms File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in from .bbox_nms import multiclass_nms from .bbox_nms import multiclass_nms File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in from .bbox_nms import multiclass_nms File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in from .bbox_nms import multiclass_nms from mmdet.ops.nms import nms_wrapper File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/core/post_processing/bbox_nms.py", line 3, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in from mmdet.ops.nms import nms_wrapper from mmdet.ops.nms import nms_wrapper from mmdet.ops.nms import nms_wrapper File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in from mmdet.ops.nms import nms_wrapper File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in from mmdet.ops.nms import nms_wrapper File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in from mmdet.ops.nms import nms_wrapper File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in from .roi_align import RoIAlign, roi_align File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in from mmdet.ops.nms import nms_wrapper File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/init.py", line 2, in from .roi_align import RoIAlign, roi_align File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in from .roi_align import RoIAlign, roi_align File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in from .roi_align import RoIAlign, roi_align File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in from .roi_align import RoIAlign, roi_align from .roi_align import RoIAlign, roi_align from .roi_align import RoIAlign, roi_align File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in from .functions.roi_align import roi_align File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in from .roi_align import RoIAlign, roi_align from .functions.roi_align import roi_align File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/init.py", line 1, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in from .functions.roi_align import roi_align from .functions.roi_align import roi_align File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in from .functions.roi_align import roi_align from .functions.roi_align import roi_align File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in from .functions.roi_align import roi_align File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in from .. import roi_align_cuda ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE from .. import roi_align_cuda from .functions.roi_align import roi_align ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE from .. import roi_align_cuda from .. import roi_align_cuda from .. import roi_align_cuda ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE File "/home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/functions/roi_align.py", line 3, in from .. import roi_align_cuda ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE from .. import roi_align_cuda from .. import roi_align_cuda ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE ImportError: /home/madongliang/.conda/envs/py36/lib/python3.6/site-packages/mmdet/ops/roi_align/roi_align_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN2at15UndefinedTensor10_singletonE

I`m using CUDA 9.0, pytorch 1.0.1.post2, python 3.6 Everything compiled well during installation.

Thanks!

[libuyu]

According to my experience, this error happens when the cuda tools are not linked properly in the conda environment or the cuda version is not compatible with the code. I used to solve this problem by creating a totally new conda environment. For more suggestions, you can refer to https://github.com/open-mmlab/mmdetection/issues/385 .

[ruiyuanlu]

Try

import torch

at the very first in your code. If it doesn't work, maybe this comment can help you.

[Vipermdl]

According to my experience, this error happens when the cuda tools are not linked properly in the conda environment or the cuda version is not compatible with the code. I used to solve this problem by creating a totally new conda environment. For more suggestions, you can refer to open-mmlab/mmdetection#385 .

thanks, this is working.