GPTQ-for-LLaMa
GPTQ-for-LLaMa copied to clipboard
qwopqwop200
The detected CUDA version (12.0) mismatches the version that was used to compile PyTorch (11.7)
Hello,
First of all, thank you for making this.
The error happens when I'm trying to execute setup_cuda.py.
Is it possible for you (or anyone) to compile the same library using newer CUDA?
(textgen) [sadmin@sherper GPTQ-for-LLaMa]$ python setup_cuda.py install
running install
/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarni
ng: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecati
onWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running bdist_egg
running egg_info
creating quant_cuda.egg-info
writing quant_cuda.egg-info/PKG-INFO
writing dependency_links to quant_cuda.egg-info/dependency_links.txt
writing top-level names to quant_cuda.egg-info/top_level.txt
writing manifest file 'quant_cuda.egg-info/SOURCES.txt'
/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/utils/cpp_extension.py:476: UserWarning: Attempted to
use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
reading manifest file 'quant_cuda.egg-info/SOURCES.txt'
writing manifest file 'quant_cuda.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
writing manifest file 'quant_cuda.egg-info/SOURCES.txt' [18/145]
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
Traceback (most recent call last):
File "/home/sadmin/services/repositories/GPTQ-for-LLaMa/setup_cuda.py", line 4, in <module>
setup(
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
return distutils.core.setup(**attrs)
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_com
mands
dist.run_commands()
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_com
mands
self.run_command(cmd)
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
super().run_command(command)
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_com
mand
cmd_obj.run()
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/command/install.py", line 74, in run
self.do_egg_install()
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/command/install.py", line 123, in do_egg_
install
self.run_command('bdist_egg')
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_comm
and
self.distribution.run_command(command)
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
super().run_command(command)
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_com
mand
cmd_obj.run()
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/command/bdist_egg.py", line 165, in run
cmd = self.call_command('install_lib', warn_dir=0)
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/command/bdist_egg.py", line 151, in call_
command
self.run_command(cmdname)
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_comm
and
self.distribution.run_command(command)
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
super().run_command(command)
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_com
mand
cmd_obj.run()
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/command/install_lib.py", line 11, in run
self.build()
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/command/install_lib.py", line
112, in build
self.run_command('build_ext')
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_comm
and
self.distribution.run_command(command)
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
super().run_command(command)
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_com
mand
cmd_obj.run()
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 84, in run
_build_ext.run(self)
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 34
6, in run
self.build_extensions()
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 499, in build_ex
tensions
_check_cuda_version(compiler_name, compiler_version)
File "/home/sadmin/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 386, in _check_c
uda_version
raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
RuntimeError:
The detected CUDA version (12.0) mismatches the version that was used to compile
PyTorch (11.7). Please make sure to use the same CUDA versions.
just install 11.7 cuda and symlink /usr/local/cuda-11-7 it to /usr/local/cuda
just install 11.7 cuda and symlink /usr/local/cuda-11-7 it to /usr/local/cuda
That unfortunately won't work, because my OS (Fedora 37) doesn't support gcc++ 11, only 12.. So even if I do a workaround and install 11.7 CUDA (I tried it) it won't launch, since it requires gcc++ 11, not 12.
Now the next workaround is to manually build gcc++ from source, but it's not really advised.
Following the installation guide from zero ends with this, when trying to run setup_cuda.py.:
subprocess.CalledProcessError: Command '['which', 'g++']' returned non-zero exit status 1.
just install 11.7 cuda and symlink /usr/local/cuda-11-7 it to /usr/local/cuda
That unfortunately won't work, because my OS (Fedora 37) doesn't support gcc++ 11, only 12.. So even if I do a workaround and install 11.7 CUDA (I tried it) it won't launch, since it requires gcc++ 11, not 12.
Now the next workaround is to manually build gcc++ from source, but it's not really advised.
Could you chroot a debian system? Not ideal, but could be reasonable for compiling binaries.
I ended up creating an ubuntu 22.04 virtual machine and using vfio GPU passthrough.
That unfortunately won't work, because my OS (Fedora 37) doesn't support gcc++ 11, only 12.. So even if I do a workaround and install 11.7 CUDA (I tried it) it won't launch, since it requires gcc++ 11, not 12.
Install LinuxBrew, then brew install gcc-11. You can then use the CC and CXX environment variables to point to the /home/linuxbrew/.linuxbrew/bin/ executables. Worked for me on Fedora 37.
same issue here, archlinux, cuda 12.1. Is VM really the only option?
