AdelaiDet icon indicating copy to clipboard operation
AdelaiDet copied to clipboard

Published 20 hours ago • verified publisher icon aim-uofa

running setup.py with some problem /bin/sh: 1: :/usr/local/cuda/bin/nvcc: not found

Open Pichenze opened this issue 3 years ago • 3 comments

my environment is
CUDA version : 10.1 NVIDIA (R) Cuda compiler driver :10.1 python version : 3.7.9 torch version : 1.6.0 gcc version : 5.4.0

I was running python setup.py build develop

and get this problem:

running build running build_py running build_ext building 'adet.C' extension Emitting ninja build file /home/ww/hdd/czy/AdelaiDet/build/temp.linux-x86_64-3.7/build.ninja... Compiling objects... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) [1/4] :/usr/local/cuda/bin/nvcc -DWITH_CUDA -I/home/ww/hdd/czy/AdelaiDet/adet/layers/csrc -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/TH -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/THC -I:/usr/local/cuda/include -I/home/ww/hdd/minicao/envs/detectron2/include/python3.7m -c -c /home/ww/hdd/czy/AdelaiDet/adet/layers/csrc/ml_nms/ml_nms.cu -o /home/ww/hdd/czy/AdelaiDet/build/temp.linux-x86_64-3.7/home/ww/hdd/czy/AdelaiDet/adet/layers/csrc/ml_nms/ml_nms.o -D__CUDA_NO_HALF_OPERATORS_ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -std=c++14 FAILED: /home/ww/hdd/czy/AdelaiDet/build/temp.linux-x86_64-3.7/home/ww/hdd/czy/AdelaiDet/adet/layers/csrc/ml_nms/ml_nms.o :/usr/local/cuda/bin/nvcc -DWITH_CUDA -I/home/ww/hdd/czy/AdelaiDet/adet/layers/csrc -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/TH -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/THC -I:/usr/local/cuda/include -I/home/ww/hdd/minicao/envs/detectron2/include/python3.7m -c -c /home/ww/hdd/czy/AdelaiDet/adet/layers/csrc/ml_nms/ml_nms.cu -o /home/ww/hdd/czy/AdelaiDet/build/temp.linux-x86_64-3.7/home/ww/hdd/czy/AdelaiDet/adet/layers/csrc/ml_nms/ml_nms.o -D__CUDA_NO_HALF_OPERATORS_ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -std=c++14 /bin/sh: 1: :/usr/local/cuda/bin/nvcc: not found [2/4] :/usr/local/cuda/bin/nvcc -DWITH_CUDA -I/home/ww/hdd/czy/AdelaiDet/adet/layers/csrc -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/TH -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/THC -I:/usr/local/cuda/include -I/home/ww/hdd/minicao/envs/detectron2/include/python3.7m -c -c /home/ww/hdd/czy/AdelaiDet/adet/layers/csrc/DefROIAlign/DefROIAlign_cuda.cu -o /home/ww/hdd/czy/AdelaiDet/build/temp.linux-x86_64-3.7/home/ww/hdd/czy/AdelaiDet/adet/layers/csrc/DefROIAlign/DefROIAlign_cuda.o -D__CUDA_NO_HALF_OPERATORS_ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -std=c++14 FAILED: /home/ww/hdd/czy/AdelaiDet/build/temp.linux-x86_64-3.7/home/ww/hdd/czy/AdelaiDet/adet/layers/csrc/DefROIAlign/DefROIAlign_cuda.o :/usr/local/cuda/bin/nvcc -DWITH_CUDA -I/home/ww/hdd/czy/AdelaiDet/adet/layers/csrc -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/TH -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/THC -I:/usr/local/cuda/include -I/home/ww/hdd/minicao/envs/detectron2/include/python3.7m -c -c /home/ww/hdd/czy/AdelaiDet/adet/layers/csrc/DefROIAlign/DefROIAlign_cuda.cu -o /home/ww/hdd/czy/AdelaiDet/build/temp.linux-x86_64-3.7/home/ww/hdd/czy/AdelaiDet/adet/layers/csrc/DefROIAlign/DefROIAlign_cuda.o -D__CUDA_NO_HALF_OPERATORS_ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -std=c++14 /bin/sh: 1: :/usr/local/cuda/bin/nvcc: not found [3/4] :/usr/local/cuda/bin/nvcc -DWITH_CUDA -I/home/ww/hdd/czy/AdelaiDet/adet/layers/csrc -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/TH -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/THC -I:/usr/local/cuda/include -I/home/ww/hdd/minicao/envs/detectron2/include/python3.7m -c -c /home/ww/hdd/czy/AdelaiDet/adet/layers/csrc/BezierAlign/BezierAlign_cuda.cu -o /home/ww/hdd/czy/AdelaiDet/build/temp.linux-x86_64-3.7/home/ww/hdd/czy/AdelaiDet/adet/layers/csrc/BezierAlign/BezierAlign_cuda.o -D__CUDA_NO_HALF_OPERATORS_ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -std=c++14 FAILED: /home/ww/hdd/czy/AdelaiDet/build/temp.linux-x86_64-3.7/home/ww/hdd/czy/AdelaiDet/adet/layers/csrc/BezierAlign/BezierAlign_cuda.o :/usr/local/cuda/bin/nvcc -DWITH_CUDA -I/home/ww/hdd/czy/AdelaiDet/adet/layers/csrc -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/TH -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/THC -I:/usr/local/cuda/include -I/home/ww/hdd/minicao/envs/detectron2/include/python3.7m -c -c /home/ww/hdd/czy/AdelaiDet/adet/layers/csrc/BezierAlign/BezierAlign_cuda.cu -o /home/ww/hdd/czy/AdelaiDet/build/temp.linux-x86_64-3.7/home/ww/hdd/czy/AdelaiDet/adet/layers/csrc/BezierAlign/BezierAlign_cuda.o -D__CUDA_NO_HALF_OPERATORS_ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -std=c++14 /bin/sh: 1: :/usr/local/cuda/bin/nvcc: not found [4/4] :/usr/local/cuda/bin/nvcc -DWITH_CUDA -I/home/ww/hdd/czy/AdelaiDet/adet/layers/csrc -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/TH -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/THC -I:/usr/local/cuda/include -I/home/ww/hdd/minicao/envs/detectron2/include/python3.7m -c -c /home/ww/hdd/czy/AdelaiDet/adet/layers/csrc/cuda_version.cu -o /home/ww/hdd/czy/AdelaiDet/build/temp.linux-x86_64-3.7/home/ww/hdd/czy/AdelaiDet/adet/layers/csrc/cuda_version.o -D__CUDA_NO_HALF_OPERATORS_ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -std=c++14 FAILED: /home/ww/hdd/czy/AdelaiDet/build/temp.linux-x86_64-3.7/home/ww/hdd/czy/AdelaiDet/adet/layers/csrc/cuda_version.o :/usr/local/cuda/bin/nvcc -DWITH_CUDA -I/home/ww/hdd/czy/AdelaiDet/adet/layers/csrc -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/TH -I/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/include/THC -I:/usr/local/cuda/include -I/home/ww/hdd/minicao/envs/detectron2/include/python3.7m -c -c /home/ww/hdd/czy/AdelaiDet/adet/layers/csrc/cuda_version.cu -o /home/ww/hdd/czy/AdelaiDet/build/temp.linux-x86_64-3.7/home/ww/hdd/czy/AdelaiDet/adet/layers/csrc/cuda_version.o -D__CUDA_NO_HALF_OPERATORS_ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -std=c++14 /bin/sh: 1: :/usr/local/cuda/bin/nvcc: not found ninja: build stopped: subcommand failed. Traceback (most recent call last): File "/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1515, in _run_ninja_build env=env) File "/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/subprocess.py", line 512, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "setup.py", line 92, in cmdclass={"build_ext": torch.utils.cpp_extension.BuildExtension}, File "/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/setuptools/init.py", line 153, in setup return distutils.core.setup(**attrs) File "/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/distutils/core.py", line 148, in setup dist.run_commands() File "/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/distutils/dist.py", line 966, in run_commands self.run_command(cmd) File "/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/distutils/command/build.py", line 135, in run self.run_command(cmd_name) File "/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 79, in run _build_ext.run(self) File "/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run _build_ext.build_ext.run(self) File "/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/distutils/command/build_ext.py", line 340, in run self.build_extensions() File "/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 649, in build_extensions build_ext.build_extensions(self) File "/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions _build_ext.build_ext.build_extensions(self) File "/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions self._build_extensions_serial() File "/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial self.build_extension(ext) File "/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 196, in build_extension _build_ext.build_extension(self, ext) File "/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/distutils/command/build_ext.py", line 534, in build_extension depends=ext.depends) File "/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 478, in unix_wrap_ninja_compile with_cuda=with_cuda) File "/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1233, in _write_ninja_file_and_compile_objects error_prefix='Error compiling objects for extension') File "/home/ww/hdd/minicao/envs/detectron2/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1529, in _run_ninja_build raise RuntimeError(message) RuntimeError: Error compiling objects for extension

Is the problem of nvcc or ninja?How can I solve this?

Pichenze avatar Apr 02 '21 11:04 Pichenze

As it says in your log: /bin/sh: 1: :/usr/local/cuda/bin/nvcc: not found. This probably means that CUDA either isn't installed, or that it isn't added to your path. Running ubuntu, this can be solved by checking where your cuda installation lives (usually somewhere like /usr/local/cuda) and adding "export PATH=/usr/local/cuda-VERSION/bin:$PATH" to your .bashrc file.

SuijkerbuijkP avatar Apr 10 '21 13:04 SuijkerbuijkP

Same issue. You can try export CUDA_HOME=/usr/local/cuda It works for me

Tianjin3 avatar Apr 13 '21 07:04 Tianjin3

@Tianjin3 it works! when you fail about something error of nvcc, you can do environment variable of cuda again, then add CUDA_HOME=/usr/local/cuda.

command: vim ~/.bashrc

add at last(this include environment variable): CUDA_HOME=/usr/local/cuda export PATH="/usr/local/cuda/bin:$PATH" export LD_LIBRARY_PATH="/usr/local/cuda/lib64:$LD_LIBRARY_PATH"

source ~/.bashrc

you should make it now.

songyangsun avatar Apr 13 '22 11:04 songyangsun