GPTQ-for-LLaMa
GPTQ-for-LLaMa copied to clipboard
qwopqwop200
Error when installing cuda kernel
If I follow the instructions in the readme, I'm getting an error now even though it worked a few days ago.
conda create --name gptq python=3.9 -y
conda activate gptq
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
git clone https://github.com/qwopqwop200/GPTQ-for-LLaMa
cd GPTQ-for-LLaMa
pip install -r requirements.txt
python setup_cuda.py install
Output:
raceback (most recent call last):
File "~/text-generation-webui/repositories/GPTQ-for-LLaMa/setup_cuda.py", line 6, in <module>
ext_modules=[cpp_extension.CUDAExtension(
File "~/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1048, in CUDAExtension
library_dirs += library_paths(cuda=True)
File "~/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1179, in library_paths
if (not os.path.exists(_join_cuda_home(lib_dir)) and
File "~/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 2223, in _join_cuda_home
raise EnvironmentError('CUDA_HOME environment variable is not set. '
OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
If I try to manually set CUDA_HOME=$CONDA_PREFIX/ (which wasn't necessary previously) it still doesn't work. I get this error:
running install
~/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
~/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running bdist_egg
running egg_info
writing quant_cuda.egg-info/PKG-INFO
writing dependency_links to quant_cuda.egg-info/dependency_links.txt
writing top-level names to quant_cuda.egg-info/top_level.txt
reading manifest file 'quant_cuda.egg-info/SOURCES.txt'
writing manifest file 'quant_cuda.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
error: [Errno 2] No such file or directory: 'CUDA_HOME=~/miniconda3/envs/textgen/bin/nvcc'
The following seems to work for me:
# ... as before
cd GPTQ-for-LLaMa
pip install -r requirements.txt
# Add the following line:
conda install -c conda-forge cudatoolkit-dev
python setup_cuda.py install
The following seems to work for me:
# ... as before cd GPTQ-for-LLaMa pip install -r requirements.txt # Add the following line: conda install -c conda-forge cudatoolkit-dev python setup_cuda.py install
This actually does work but I had trouble getting conda install -c conda-forge cudatoolkit-dev to complete successfully because my /tmp was running out of space. I had to increase the max amount of RAM that tmpfs would use for /tmp using this:
sudo mount -o remount,size=12G /tmp
https://wiki.archlinux.org/title/tmpfs
mateusz's command gets stuck on
Preparing transaction: done
Verifying transaction: done
Executing transaction: -
for me. Something changed in pytorch's official package for conda/pip and it seems like nvcc is no longer being included.
mateusz's command gets stuck on
Preparing transaction: done Verifying transaction: done Executing transaction: -
Seems like it just takes a long time running a wget command silently downloading from nvidia
After it eventally completes for me, Im getting:
RuntimeError:
The detected CUDA version (12.1) mismatches the version that was used to compile
PyTorch (11.7). Please make sure to use the same CUDA versions.
[ #13 ] $ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_May__3_18:49:52_PDT_2022
Cuda compilation tools, release 11.7, V11.7.64
Build cuda_11.7.r11.7/compiler.31294372_0
@qwopqwop200 i'm getting RuntimeError: The detected CUDA version (12.1) mismatches the version that was used to compile PyTorch (11.7). Please make sure to use the same CUDA versions. , how to fix this?
