[Bug] RuntimeError: failed to create detector

[Daanfb]

Checklist

• [X] I have searched related issues but cannot get the expected help.
• [X] 2. I have read the FAQ documentation but cannot get the expected help.
• [X] 3. The bug has not been fixed in the latest version.

Describe the bug

I have exported the model with this command:

python tools/deploy.py
configs/mmdet/detection/detection_onnxruntime_dynamic.py
../mmpose/projects/rtmpose/rtmdet/person/rtmdet_nano_320-8xb32_coco-person.py
https://download.openmmlab.com/mmpose/v1/projects/rtmpose/rtmdet_nano_8xb32-100e_coco-obj365-person-05d8511e.pth
demo/resources/human-pose.jpg
--work-dir rtmpose-ort/rtmdet-nano
--device cpu
--show
--dump-info # dump sdk info

In my python code, if i put device='cuda' i get an error, but if I put device='cpu' it works well. How can I use cuda? I tried to execute with --device cuda but I get an error too.

The route indicated (rtmpose-ort/rtmdet-nano) is a directory with the .onnx and .json files

Reproduction

from mmdeploy_runtime import Detector

det_model_path = 'rtmpose-ort/rtmdet-nano' device_name = 'cuda'

bbox_detector = Detector(det_model_path, device_name, device_id=0)

Environment

03/04 12:22:30 - mmengine - INFO -

03/04 12:22:30 - mmengine - INFO - **********Environmental information**********
03/04 12:22:35 - mmengine - INFO - sys.platform: win32
03/04 12:22:35 - mmengine - INFO - Python: 3.8.18 (default, Sep 11 2023, 13:39:12) [MSC v.1916 64 bit (AMD64)]
03/04 12:22:35 - mmengine - INFO - CUDA available: True
03/04 12:22:35 - mmengine - INFO - MUSA available: False
03/04 12:22:35 - mmengine - INFO - numpy_random_seed: 2147483648
03/04 12:22:35 - mmengine - INFO - GPU 0: NVIDIA GeForce GTX 1650
03/04 12:22:35 - mmengine - INFO - CUDA_HOME: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8
03/04 12:22:35 - mmengine - INFO - NVCC: Cuda compilation tools, release 11.8, V11.8.89
03/04 12:22:35 - mmengine - INFO - MSVC: Compilador de optimización de C/C++ de Microsoft (R) versión 19.39.33520 para x64
03/04 12:22:35 - mmengine - INFO - GCC: n/a
03/04 12:22:35 - mmengine - INFO - PyTorch: 2.0.1
03/04 12:22:35 - mmengine - INFO - PyTorch compiling details: PyTorch built with:
  - C++ Version: 199711
  - MSVC 193431937
  - Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)
  - OpenMP 2019
  - LAPACK is enabled (usually provided by MKL)
  - CPU capability usage: AVX2
  - CUDA Runtime 11.8
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_37,code=compute_37
  - CuDNN 8.7
  - Magma 2.5.4
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=C:/cb/pytorch_1000000000000/work/tmp_bin/sccache-cl.exe, CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj /FS -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=OFF, TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF,

03/04 12:22:35 - mmengine - INFO - TorchVision: 0.15.2
03/04 12:22:35 - mmengine - INFO - OpenCV: 4.9.0
03/04 12:22:35 - mmengine - INFO - MMEngine: 0.10.3
03/04 12:22:35 - mmengine - INFO - MMCV: 2.1.0
03/04 12:22:35 - mmengine - INFO - MMCV Compiler: MSVC 192930148
03/04 12:22:35 - mmengine - INFO - MMCV CUDA Compiler: 11.8
03/04 12:22:35 - mmengine - INFO - MMDeploy: 1.3.1+b30ba35
03/04 12:22:35 - mmengine - INFO -

03/04 12:22:35 - mmengine - INFO - **********Backend information**********
03/04 12:22:35 - mmengine - INFO - tensorrt:    None
03/04 12:22:36 - mmengine - INFO - ONNXRuntime: 1.8.1
03/04 12:22:36 - mmengine - INFO - ONNXRuntime-gpu:     None
03/04 12:22:36 - mmengine - INFO - ONNXRuntime custom ops:      NotAvailable
03/04 12:22:36 - mmengine - INFO - pplnn:       None
03/04 12:22:36 - mmengine - INFO - ncnn:        None
03/04 12:22:36 - mmengine - INFO - snpe:        None
03/04 12:22:36 - mmengine - INFO - openvino:    None
03/04 12:22:36 - mmengine - INFO - torchscript: 2.0.1
03/04 12:22:36 - mmengine - INFO - torchscript custom ops:      NotAvailable
03/04 12:22:36 - mmengine - INFO - rknn-toolkit:        None
03/04 12:22:36 - mmengine - INFO - rknn-toolkit2:       None
03/04 12:22:36 - mmengine - INFO - ascend:      None
03/04 12:22:36 - mmengine - INFO - coreml:      None
03/04 12:22:36 - mmengine - INFO - tvm: None
03/04 12:22:36 - mmengine - INFO - vacc:        None
03/04 12:22:36 - mmengine - INFO -

03/04 12:22:36 - mmengine - INFO - **********Codebase information**********
03/04 12:22:36 - mmengine - INFO - mmdet:       3.2.0
03/04 12:22:36 - mmengine - INFO - mmseg:       None
03/04 12:22:36 - mmengine - INFO - mmpretrain:  None
03/04 12:22:36 - mmengine - INFO - mmocr:       None
03/04 12:22:36 - mmengine - INFO - mmagic:      None
03/04 12:22:36 - mmengine - INFO - mmdet3d:     None
03/04 12:22:36 - mmengine - INFO - mmpose:      1.3.1
03/04 12:22:36 - mmengine - INFO - mmrotate:    None
03/04 12:22:36 - mmengine - INFO - mmaction:    None
03/04 12:22:36 - mmengine - INFO - mmrazor:     None
03/04 12:22:36 - mmengine - INFO - mmyolo:      None

Error traceback

RuntimeError                              Traceback (most recent call last)
Cell In[26], line 1
----> 1 bbox_detector = Detector(det_model_path, device_name, device_id=0)

RuntimeError: failed to create detector

[cmc9595]

@Daanfb Hi, I have exact same situation here, have you found any solution yet? Thanks in advance.

[LLH-Harward]

@Daanfb @cmc9595 Hi, I have the same problem too. Has there been any progress? Thank you.

[LLH-Harward]

First, ensure that there are no onnxruntime or mmdeploy-runtime files on your computer. Delete all such files. Then, install onnxruntime-gpu, mmdeploy, and mmdeploy-runtime-gpu (they cannot coexist).

Install the corresponding versions of TensorRT and cuDNN. After installing ONNX Runtime, everything should work fine. I have installed CUDA 11.8 and ONNX Runtime GPU 11.6.

Additionally,Warning: [W] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage and speed up TensorRT initialization. See "Lazy Loading" section of CUDA documentation https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#lazy-loading. Solution: set CUDA_MODULE_LOADING=LAZY

[cmc9595]

@LLH-Harward Thank you for your reply. I also found https://mmdeploy.readthedocs.io/en/latest/04-supported-codebases/mmocr.html#backend-model-inference backend model inference code works fine for me, so I'm trying to combine this code with Triton inference server first. If I fail, I would have to return back to https://mmdeploy.readthedocs.io/en/latest/04-supported-codebases/mmocr.html#sdk-model-inference SDK Inference code using mmdeploy_runtime, so..

[LLH-Harward]

@cmc9595 Thank you for your reply! Please let me know if there are any updates.