[Feature] Implement nms_impl on macOS MPS device

[ypwhs]

What is the feature?

When running detection models on my MacBook, if I use MPS inference, it throws an error stating that nms has not been implemented. I am hoping that it can be implemented.

Traceback:

Traceback (most recent call last):
  File "/Users/ypw/Desktop/detection/detection/detect/predictor.py", line 295, in predict
    inference_result = inference(
  File "/Users/ypw/Desktop/detection/detection/detect/utils_torchvision.py", line 234, in inference
    results = model(return_loss=False, rescale=True, **data)
  File "/Users/ypw/miniconda3_arm64/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/ypw/miniconda3_arm64/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 119, in new_func
    return old_func(*args, **kwargs)
  File "/Users/ypw/miniconda3_arm64/lib/python3.8/site-packages/mmdet/models/detectors/base.py", line 174, in forward
    return self.forward_test(img, img_metas, **kwargs)
  File "/Users/ypw/miniconda3_arm64/lib/python3.8/site-packages/mmdet/models/detectors/base.py", line 147, in forward_test
    return self.simple_test(imgs[0], img_metas[0], **kwargs)
  File "/Users/ypw/miniconda3_arm64/lib/python3.8/site-packages/mmdet/models/detectors/two_stage.py", line 179, in simple_test
    proposal_list = self.rpn_head.simple_test_rpn(x, img_metas)
  File "/Users/ypw/miniconda3_arm64/lib/python3.8/site-packages/mmdet/models/dense_heads/dense_test_mixins.py", line 130, in simple_test_rpn
    proposal_list = self.get_bboxes(*rpn_outs, img_metas=img_metas)
  File "/Users/ypw/miniconda3_arm64/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 208, in new_func
    return old_func(*args, **kwargs)
  File "/Users/ypw/miniconda3_arm64/lib/python3.8/site-packages/mmdet/models/dense_heads/base_dense_head.py", line 102, in get_bboxes
    results = self._get_bboxes_single(cls_score_list, bbox_pred_list,
  File "/Users/ypw/miniconda3_arm64/lib/python3.8/site-packages/mmdet/models/dense_heads/rpn_head.py", line 185, in _get_bboxes_single
    return self._bbox_post_process(mlvl_scores, mlvl_bbox_preds,
  File "/Users/ypw/miniconda3_arm64/lib/python3.8/site-packages/mmdet/models/dense_heads/rpn_head.py", line 231, in _bbox_post_process
    dets, _ = batched_nms(proposals, scores, ids, cfg.nms)
  File "/Users/ypw/miniconda3_arm64/lib/python3.8/site-packages/mmcv/ops/nms.py", line 350, in batched_nms
    dets, keep = nms_op(boxes_for_nms, scores, **nms_cfg_)
  File "/Users/ypw/miniconda3_arm64/lib/python3.8/site-packages/mmcv/utils/misc.py", line 340, in new_func
    output = old_func(*args, **kwargs)
  File "/Users/ypw/miniconda3_arm64/lib/python3.8/site-packages/mmcv/ops/nms.py", line 175, in nms
    inds = NMSop.apply(boxes, scores, iou_threshold, offset, score_threshold,
  File "/Users/ypw/miniconda3_arm64/lib/python3.8/site-packages/mmcv/ops/nms.py", line 28, in forward
    inds = ext_module.nms(
RuntimeError: nms_impl: implementation for device mps:0 not found.

Any other context?

Environment info:

/Users/ypw/miniconda3_arm64/lib/python3.8/site-packages/mmcv/__init__.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
  warnings.warn(
/Users/ypw/miniconda3_arm64/lib/python3.8/runpy.py:127: RuntimeWarning: 'mmdet.utils.collect_env' found in sys.modules after import of package 'mmdet.utils', but prior to execution of 'mmdet.utils.collect_env'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
sys.platform: darwin
Python: 3.8.11 (default, Jul 29 2021, 14:57:32) [Clang 12.0.0 ]
CUDA available: False
GCC: Apple clang version 14.0.3 (clang-1403.0.22.14.1)
PyTorch: 1.13.1
PyTorch compiling details: PyTorch built with:
  - GCC 4.2
  - C++ Version: 201402
  - clang 14.0.0
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: NO AVX
  - Build settings: BLAS_INFO=accelerate, BUILD_TYPE=Release, CXX_COMPILER=/Applications/Xcode_14.0.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -Wno-deprecated-declarations -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_PYTORCH_METAL_EXPORT -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -DUSE_COREML_DELEGATE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wvla-extension -Wno-range-loop-analysis -Wno-pass-failed -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -Wconstant-conversion -Wno-invalid-partial-specialization -Wno-typedef-redefinition -Wno-unused-private-field -Wno-inconsistent-missing-override -Wno-c++14-extensions -Wno-constexpr-not-const -Wno-missing-braces -Wunused-lambda-capture -Wunused-local-typedef -Qunused-arguments -fcolor-diagnostics -fdiagnostics-color=always -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -DUSE_MPS -fno-objc-arc -Wno-unguarded-availability-new -Wno-unused-private-field -Wno-missing-braces -Wno-c++14-extensions -Wno-constexpr-not-const, LAPACK_INFO=accelerate, TORCH_VERSION=1.13.1, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EIGEN_FOR_BLAS=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=OFF, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=OFF, USE_ROCM=OFF,

TorchVision: 0.14.1
OpenCV: 4.7.0
MMCV: 1.7.1
MMCV Compiler: clang 14.0.3
MMCV CUDA Compiler: not available
MMDetection: 2.28.2+bfdd1f9

[zhouzaida]

Hi @ypwhs , sorry for my late reply, we hope to have complete support for MPS operators, but we don't have enough time. Supported ops can be found at https://mmcv.readthedocs.io/en/latest/understand_mmcv/ops.html.

[pa-pallon]

Is there a way to disable MPS on mac M1 and use the CPU to invoke NMS?

[zhouzaida]

Is there a way to disable MPS on mac M1 and use the CPU to invoke NMS?

Hi, if mps is available, the mps ops will be compiled. https://github.com/open-mmlab/mmcv/blob/3269278ed0ef2f340bb8d5fc3334e85c08139e1d/setup.py#L338-L340

If you want to use the CPU implementation, you need to make sure the data is on CPU and the CPU implementation will be called.

[pa-pallon]

Thanks @zhouzaida I have installed the CPU wheel (https://download.openmmlab.com/mmcv/dist/cpu/torch2.0.0/mmcv-2.0.0-cp310-cp310-macosx_10_15_x86_64.whl) but my data is on MPS.

I find that that mmengine DataPreprocessor moves the data to the target device . I would like to run mmdetection repo on CUDA and M1. The get_device() return mlu on Mac. Is there a way to smoothly transfer data to CPU when running on Mac?