Error while trying to install nms-extension

[pereaanamaria]

Hello!

I am trying to install the custom extensions from MDT_ProstateX: custom_extensions/nms, custom_extensions/roi_align/2D, and custom_extensions/roi_align/3D. RoIAlign-extension-2D and RoIAlign-extension-3D have been successfully built, but nms-extension keeps failing. I have attached below the console output I get everytime I am trying to run setup.py.

I am using PyCharm Community Edition 2021.2.3 on Windows 10, with NVIDIA GPU Computing Toolkit/CUDA/v11.6 and Visual Studio Build Tools 2019. The virtual environment uses Python 3.7, as suggested in the README.md document.

  building 'nms_extension' extension
  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Users\anama\anaconda3\envs\prostate_lesion\lib\site-packages\torch\include -IC:\Users\anama\anaconda3\envs\prostate_lesion\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\anama\anaconda3\envs\prostate_lesion\lib\site-packages\torch\include\TH -IC:\Users\anama\anaconda3\envs\prostate_lesion\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\include" -IC:\Users\anama\anaconda3\envs\prostate_lesion\include -IC:\Users\anama\anaconda3\envs\prostate_lesion\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" /EHsc /TpD:\git\prostate_lesion_detection\MDT_ProstateX\custom_extensions\nms\src\nms_interface.cpp /Fobuild\temp.win-amd64-3.7\Release\git\prostate_lesion_detection\MDT_ProstateX\custom_extensions\nms\src\nms_interface.obj /MD /wd4819 /EHsc -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=nms_extension -D_GLIBCXX_USE_CXX11_ABI=0
  nms_interface.cpp
  C:\Users\anama\anaconda3\envs\prostate_lesion\lib\site-packages\torch\include\c10/util/Exception.h(28): warning C4275: non dll-interface class 'std::exception' used as base for dll-interface class 'c10::Error'
  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\include\vcruntime_exception.h(48): note: see declaration of 'std::exception'
  C:\Users\anama\anaconda3\envs\prostate_lesion\lib\site-packages\torch\include\c10/util/Exception.h(28): note: see declaration of 'c10::Error'
  C:\Users\anama\anaconda3\envs\prostate_lesion\lib\site-packages\torch\include\c10/util/Exception.h(29): warning C4251: 'c10::Error::msg_stack_': class 'std::vector<std::string,std::allocator<std::string>>' needs to have dll-interface to be used by clients of class 'c10::Error'
  C:\Users\anama\anaconda3\envs\prostate_lesion\lib\site-packages\torch\include\c10/util/Exception.h(29): note: see declaration of 'std::vector<std::string,std::allocator<std::string>>'
  C:\Users\anama\anaconda3\envs\prostate_lesion\lib\site-packages\torch\include\c10/util/Exception.h(30): warning C4251: 'c10::Error::backtrace_': class 'std::basic_string<char,std::char_traits<char>,std::allocator<char>>' needs to have dll-interface to be used by clients of class 'c10::Error'
 
  [Many warnings listed from C:\Users\anama\anaconda3\envs\prostate_lesion\lib\site-packages\torch\include: warning C4251, C4273, C4275, C4267, C4244 etc.]

  C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\bin\nvcc.exe -c D:\git\prostate_lesion_detection\MDT_ProstateX\custom_extensions\nms\src\nms.cu -o build\temp.win-amd64-3.7\Release\git\prostate_lesion_detection\MDT_ProstateX\custom_extensions\nms\src\nms.obj -IC:\Users\anama\anaconda3\envs\prostate_lesion\lib\site-packages\torch\include -IC:\Users\anama\anaconda3\envs\prostate_lesion\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\anama\anaconda3\envs\prostate_lesion\lib\site-packages\torch\include\TH -IC:\Users\anama\anaconda3\envs\prostate_lesion\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\include" -IC:\Users\anama\anaconda3\envs\prostate_lesion\include -IC:\Users\anama\anaconda3\envs\prostate_lesion\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" -Xcompiler /EHsc -Xcompiler /wd4819 -Xcompiler /MD -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=nms_extension -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_75,code=sm_75
  
  [Many warnings listed from C:\Users\anama\anaconda3\envs\prostate_lesion\lib\site-packages\torch\include: warning #1394-D, #68-D, #1388-D etc.]
  
  C:/Users/anama/anaconda3/envs/prostate_lesion/lib/site-packages/torch/include\torch/csrc/jit/script/jit_exception.h(10): warning #1388-D: base class dllexport/dllimport specification differs from that of the derived class
  C:/Users/anama/anaconda3/envs/prostate_lesion/lib/site-packages/torch/include\c10/cuda/CUDAStream.h(171): warning #1394-D: field of class type without a DLL interface used in a class with a DLL interface
  C:/Users/anama/anaconda3/envs/prostate_lesion/lib/site-packages/torch/include\TH/THAllocator.h(60): warning #1394-D: field of class type without a DLL interface used in a class with a DLL interface
  C:/Users/anama/anaconda3/envs/prostate_lesion/lib/site-packages/torch/include\TH/THAllocator.h(66): warning #1394-D: field of class type without a DLL interface used in a class with a DLL interface
  C:/Users/anama/anaconda3/envs/prostate_lesion/lib/site-packages/torch/include\THC/THCAllocator.h(14): warning #1394-D: field of class type without a DLL interface used in a class with a DLL interface
  
  2 errors detected in the compilation of "D:/git/prostate_lesion_detection/MDT_ProstateX/custom_extensions/nms/src/nms.cu".
  nms.cu
  error: command 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.6\\bin\\nvcc.exe' failed with exit status 1

I would really appreciate your help, since I am not sure what I am missing. Thank you!

[OscarPellicer]

Hi,

Unfortunately, I am not sure that I can help you. medicaldetectiontoolkit was developed for linux, and there is no support for Windows. You could try to reach the developers, but even the new framework that they have developed, nnDetection does not support Windows either.

Regards, Oscar.

[pereaanamaria]

Hi!

Thank you for the quick response. I managed to run the setup.py file successfully through WSL2 Ubuntu 20.04.4 LTS.

[OscarPellicer]

That's great to hear!

[changzhang777]

Hello, thanks for your contribution!

I have used torch1.9.0+cudnn11.4 to run

python exec.py --mode test --exp_source experiments/exp0 --exp_dir experiments/exp0

. But it brings a problem in nms.py in the line 37

return nms_extension.nms(boxes, scores, iou_threshold)

.The details are as follows.

RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Could you please give me some guidance? I will be very grateful if you can help me. Best wishes!

[OscarPellicer]

As I mentioned, WSL is not officially supported, so I can't really help you much. By searching for your error a bit online I found that you might have an older GPU, and maybe want to try with CUDA 10 instead of CUDA 11: https://stackoverflow.com/questions/67825986/nvidia-cuda-error-no-kernel-image-is-available-for-execution-on-the-device

Regards, Oscar.

[dwDavidxd]

Hello! First of all thanks for your great work. I have some confusions and I would appreciate if you could help. I noticed that the requirement.txt file in the MDT_ProstateX folder requests torch==1.4.0, torchvision==0.5.0, and you also said that you are using pytorch 1.7, can I change torch==1.4.0 , torchvision==0.5.0 replaced with torch==1.7.0, torchvision==0.8.0? Kind regards.

[OscarPellicer]

I checked my environment:

# Name                    Version                   Build  Channel
coral-pytorch             1.0.0                    pypi_0    pypi
pytorch                   1.7.0           py3.7_cuda11.0.221_cudnn8.0.3_0    pytorch
torch                     1.4.0                    pypi_0    pypi
torchaudio                0.7.0                      py37    pytorch
torchvision               0.5.0                    pypi_0    pypi

I think pytorch 1.7 is being used, and torch 1.4 is being ignored despite being installed. Also, apparently I am using torchvision 0.5 Tbh, I think it would probably be better to remove torch, and torchvision from requirements.txt and install newer versions from conda's pytorch channel. pytorch==1.7.0, torchvision==0.8.0 should very likely be fine.

[changzhang777]

Thank you for your answer ！the cause of the previous problem is indeed related to the version of the torch, I have switched to the torch 1.7 to solve the problem. But I have a new problem, when using ProstateX preprocessing.ipynb preprocessing pid=0105 samples, a warning and an error were thrown.

/home/zbx/anaconda3/envs/prostate_lesion/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3441: RuntimeWarning: Mean of empty slice.
  out=out, **kwargs)
/home/zbx/anaconda3/envs/prostate_lesion/lib/python3.7/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
 - Error: Lesion mask array is empty: no segmentations were performed

It points out that the lesion_masks_arr_list of this sample is an all-0 vector, corresponding to line 882 in the code preprocessing_lib.py: lesion_mask= np.mean([m for m in lesion_masks_arr_list if np.sum(m) > 64], axis=0) > 2.5/8 #At least 3

Is this normal and will it affect the following test and analysis?

Regards.

[OscarPellicer]

I'm pretty sure that it is normal and it should not affect later stages, but it has been a while... If you have problems later on, let me know and I will look deeper into it.