Describe the bug

i've looked at the troubleshooting posts, but perhaps i've missed something.

trying this on windows 10 for 4bit precision with 7b model

I got the regular webui running with pyg model just fine but I keep running into errors with this, different ones each time/ here's the latest error trying to use python setup_cuda.py

d:\myenvs\textgen1\lib\site-packages\torch\utils\cpp_extension.py:359: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified warnings.warn(f'Error checking compiler version for {compiler}: {error}') traceback---------------------

RuntimeError: The detected CUDA version (12.0) mismatches the version that was used to compile PyTorch (11.7). Please make sure to use the same CUDA versions.

Is there an existing issue for this?

[X] I have searched the existing issues

Reproduction

normal installation, use visual studio 2019 with desktop, pip install main webui/gptq, use the conda forge command, and then try the cuda_setup.py

Screenshot

No response

Logs

(d:\myenvs\textgen1) D:\text-generation-webui\repositories\GPTQ-for-LLaMa>python setup_cuda.py install
running install
d:\myenvs\textgen1\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
d:\myenvs\textgen1\lib\site-packages\setuptools\command\easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
running egg_info
writing quant_cuda.egg-info\PKG-INFO
writing dependency_links to quant_cuda.egg-info\dependency_links.txt
writing top-level names to quant_cuda.egg-info\top_level.txt
reading manifest file 'quant_cuda.egg-info\SOURCES.txt'
writing manifest file 'quant_cuda.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
running build_ext
d:\myenvs\textgen1\lib\site-packages\torch\utils\cpp_extension.py:359: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
  warnings.warn(f'Error checking compiler version for {compiler}: {error}')
Traceback (most recent call last):
  File "D:\text-generation-webui\repositories\GPTQ-for-LLaMa\setup_cuda.py", line 4, in <module>
    setup(
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\_distutils\core.py", line 185, in setup
    return run_commands(dist)
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands
    dist.run_commands()
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\_distutils\dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\dist.py", line 1208, in run_command
    super().run_command(command)
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
    cmd_obj.run()
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\command\install.py", line 74, in run
    self.do_egg_install()
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\command\install.py", line 123, in do_egg_install
    self.run_command('bdist_egg')
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\dist.py", line 1208, in run_command
    super().run_command(command)
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
    cmd_obj.run()
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\command\bdist_egg.py", line 165, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\command\bdist_egg.py", line 151, in call_command
    self.run_command(cmdname)
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\dist.py", line 1208, in run_command
    super().run_command(command)
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
    cmd_obj.run()
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\command\install_lib.py", line 11, in run
    self.build()
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\_distutils\command\install_lib.py", line 112, in build
    self.run_command('build_ext')
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\dist.py", line 1208, in run_command
    super().run_command(command)
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
    cmd_obj.run()
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\command\build_ext.py", line 84, in run
    _build_ext.run(self)
  File "d:\myenvs\textgen1\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 346, in run
    self.build_extensions()
  File "d:\myenvs\textgen1\lib\site-packages\torch\utils\cpp_extension.py", line 499, in build_extensions
    _check_cuda_version(compiler_name, compiler_version)
  File "d:\myenvs\textgen1\lib\site-packages\torch\utils\cpp_extension.py", line 387, in _check_cuda_version
    raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
RuntimeError:
The detected CUDA version (12.0) mismatches the version that was used to compile
PyTorch (11.7). Please make sure to use the same CUDA versions.

System Info

windows 10, nvidia 980 6gigs, local install

Mar 20 '23 18:03 gsgoldma

Default install instructions for pytorch will install 12.0 cuda files. The easiest way that I've found to get around this is to install pytorch using conda with -c "nvidia/label/cuda-11.7.0" included before -c nvidia:

conda install pytorch torchvision torchaudio pytorch-cuda=11.7 cuda-toolkit -c "nvidia/label/cuda-11.7.0" -c pytorch -c nvidia

This command includes the official cuda-toolkit install which makes the conda-forge command redundant. If you would prefer to use conda-forge, then you can remove cuda-toolkit from the above command.

Mar 20 '23 19:03 jllllll

shouldnt that be changed if its causing this error?

edit: new error after I did your command

(textgen) PS D:\text-generation-webui\repositories\gptq-for-llama> python setup_cuda.py install Traceback (most recent call last): File "D:\text-generation-webui\repositories\gptq-for-llama\setup_cuda.py", line 2, in from torch.utils import cpp_extension File "D:\miniconda3\envs\textgen\lib\site-packages\torch_init_.py", line 122, in raise err OSError: [WinError 127] The specified procedure could not be found. Error loading "D:\miniconda3\envs\textgen\lib\site-packages\torch\lib\nvfuser_codegen.dll" or one of its dependencies.

Mar 20 '23 20:03 gsgoldma

Make sure that you are on the 468c47c01b4fe370616747b6d69a2d3f48bab5e4 revision of gptq-for-llama. Run this command in the gptq-for-llama folder: git reset --hard 468c47c01b4fe370616747b6d69a2d3f48bab5e4 I just successfully compiled gptq-for-llama in Windows using VS 2022 after using that cuda-toolkit -c "nvidia/label/cuda-11.7.0" command, so I know it works.

Mar 20 '23 21:03 jllllll

i believe i already did, but to make sure, I did again and received same error.

(textgen) D:\text-generation-webui\repositories\GPTQ-for-LLaMa>git reset --hard 468c47c01b4fe370616747b6d69a2d3f48bab5e4 HEAD is now at 468c47c Update README.md

(textgen) D:\text-generation-webui\repositories\GPTQ-for-LLaMa>conda install pytorch torchvision torchaudio pytorch-cuda=11.7 cuda-toolkit -c "nvidia/label/cuda-11.7.0" -c pytorch -c nvidia Collecting package metadata (current_repodata.json): done Solving environment: done

==> WARNING: A newer version of conda exists. <== current version: 22.11.1 latest version: 23.1.0

Please update conda by running

$ conda update -n base -c defaults conda

Or to minimize the number of packages updated during conda update use

 conda install conda=23.1.0

All requested packages already installed.

(textgen) D:\text-generation-webui\repositories\GPTQ-for-LLaMa>python setup_cuda.py install Traceback (most recent call last): File "D:\text-generation-webui\repositories\GPTQ-for-LLaMa\setup_cuda.py", line 2, in from torch.utils import cpp_extension File "D:\miniconda3\envs\textgen\lib\site-packages\torch_init_.py", line 122, in raise err OSError: [WinError 127] The specified procedure could not be found. Error loading "D:\miniconda3\envs\textgen\lib\site-packages\torch\lib\nvfuser_codegen.dll" or one of its dependencies.

Mar 20 '23 21:03 gsgoldma

Conda will not re-install already existing packages, even if they are a different version. You can use --force-reinstall, but in my experience, you are better off just re-creating the environment. Also, I am using the latest version of conda, though I'm not sure that would make any difference.

Mar 20 '23 21:03 jllllll

are there any steps to avoid that you did differently?

Mar 20 '23 22:03 gsgoldma

This is the entirety of my install batch script:

@echo off

call D:\Programs\miniconda3\Scripts\activate.bat D:\Programs\miniconda3 || goto end &REM This is to prime conda for use. Will need different paths for your install.

call conda create -p %cd%\textgen pytorch torchvision torchaudio pytorch-cuda=11.7 cuda-toolkit -c "nvidia/label/cuda-11.7.0" -c pytorch -c nvidia
call conda activate %cd%\textgen

git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui || goto end
pip install -r requirements.txt

mkdir repositories
cd repositories || goto end
git clone https://github.com/qwopqwop200/GPTQ-for-LLaMa
cd GPTQ-for-LLaMa || goto end
git reset --hard 468c47c01b4fe370616747b6d69a2d3f48bab5e4
pip install -r requirements.txt

pip install ninja
call python setup_cuda.py install

:end
pause

Mar 20 '23 23:03 jllllll

Here is a pre-compiled wheel made using the environment created from the script above: quant_cuda-0.0.0-cp310-cp310-win_amd64.whl.zip It can be installed with: pip install quant_cuda-0.0.0-cp310-cp310-win_amd64.whl This should work if you can't get yours to compile. If any significant updates occur, or your conda environment is significantly different to mine, then this wheel will produce bad model output. You also may need to place the wheel file inside the GPTQ-for-LLaMa folder before installing it.

Mar 20 '23 23:03 jllllll

got a ton of errors trying that, damn

Mar 21 '23 00:03 gsgoldma

got a ton of errors trying that, damn

The batch script? Make sure to change the paths on that first call line to match your miniconda install.

Mar 21 '23 00:03 jllllll

Yeah I did replace that. But as of now, I've deleted miniconda and reinstalled, deleted my old cuda toolkit and reinstalled one from the web. now im getting permission issues with miniconda. its just a whole new shit show

running build_ext C:\WINDOWS\system32\textgen1\lib\site-packages\torch\utils\cpp_extension.py:359: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified warnings.warn(f'Error checking compiler version for {compiler}: {error}') building 'quant_cuda' extension creating build creating build\temp.win-amd64-cpython-310 creating build\temp.win-amd64-cpython-310\Release "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.34.31933\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\WINDOWS\system32\textgen1\lib\site-packages\torch\include -IC:\WINDOWS\system32\textgen1\lib\site-packages\torch\include\torch\csrc\api\include -IC:\WINDOWS\system32\textgen1\lib\site-packages\torch\include\TH -IC:\WINDOWS\system32\textgen1\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\include" -IC:\WINDOWS\system32\textgen1\include -IC:\WINDOWS\system32\textgen1\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.34.31933\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.34.31933\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\cppwinrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" /EHsc /Tpquant_cuda.cpp /Fobuild\temp.win-amd64-cpython-310\Release\quant_cuda.obj /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /EHsc -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=quant_cuda -D_GLIBCXX_USE_CXX11_ABI=0 quant_cuda.cpp c1xx: fatal error C1083: Cannot open source file: 'quant_cuda.cpp': No such file or directory error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.34.31933\bin\HostX86\x64\cl.exe' failed with exit code 2 Press any key to continue . . .

and i have no idea why its installing in win32...what a dumpster fire this has been today

Mar 21 '23 01:03 gsgoldma

That's pretty weird. It's using the x64 compiler, but it can't find quant_cuda.cpp. You should just install the wheel that I uploaded above. No clue what's going on with that. Make sure you aren't running as admin.

Mar 21 '23 01:03 jllllll

im definitely running as admin, thats what the instructions said with powershell and chatgpt told me to as well when permissions got weird.

im probably going to nuke it all and try again

and to think I get stable diffuison up and running in no time

Mar 21 '23 01:03 gsgoldma

The issue with running as admin is that it will change your current directory to your user directory before executing the script. I was able to successfully run the script without running as admin. Just put the script where you want to install the webui and double-click it.

Mar 21 '23 01:03 jllllll

"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\bin\nvcc" -c quant_cuda_kernel.cu -o build\temp.win-amd64-cpython-310\Release\quant_cuda_kernel.obj -ID:\textgen1\lib\site-packages\torch\include -ID:\textgen1\lib\site-packages\torch\include\torch\csrc\api\include -ID:\textgen1\lib\site-packages\torch\include\TH -ID:\textgen1\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\include" -ID:\textgen1\include -ID:\textgen1\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.34.31933\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.34.31933\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\cppwinrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=quant_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_52,code=compute_52 -gencode=arch=compute_52,code=sm_52 --use-local-env quant_cuda_kernel.cu D:/textgen1/lib/site-packages/torch/include\c10/macros/Macros.h(138): warning C4067: unexpected tokens following preprocessor directive - expected a newline D:/textgen1/lib/site-packages/torch/include\c10/util/irange.h(54): warning #186-D: pointless comparison of unsigned integer with zero detected during: instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator==(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=size_t, one_sided=false, =0]" (61): here instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator!=(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=size_t, one_sided=false, =0]" D:/textgen1/lib/site-packages/torch/include\c10/core/TensorImpl.h(77): here

D:/textgen1/lib/site-packages/torch/include\c10/util/irange.h(54): warning #186-D: pointless comparison of unsigned integer with zero detected during: instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator==(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=size_t, one_sided=true, =0]" (61): here instantiation of "__nv_bool c10::detail::integer_iterator<I, one_sided, >::operator!=(const c10::detail::integer_iterator<I, one_sided, > &) const [with I=size_t, one_sided=true, =0]" D:/textgen1/lib/site-packages/torch/include\ATen/core/qualified_name.h(73): here

quant_cuda_kernel.cu(149): error: no instance of overloaded function "atomicAdd" matches the argument list argument types are: (double *, double) detected during instantiation of "void VecQuant2MatMulKernel(const scalar_t *, const int *, scalar_t *, const scalar_t *, const scalar_t *, int, int, int, int) [with scalar_t=double]" (87): here

quant_cuda_kernel.cu(261): error: no instance of overloaded function "atomicAdd" matches the argument list argument types are: (double *, double) detected during instantiation of "void VecQuant3MatMulKernel(const scalar_t *, const int *, scalar_t *, const scalar_t *, const scalar_t *, int, int, int, int) [with scalar_t=double]" (171): here

quant_cuda_kernel.cu(337): error: no instance of overloaded function "atomicAdd" matches the argument list argument types are: (double *, double) detected during instantiation of "void VecQuant4MatMulKernel(const scalar_t *, const int *, scalar_t *, const scalar_t *, const scalar_t *, int, int, int, int) [with scalar_t=double]" (283): here

quant_cuda_kernel.cu(409): error: no instance of overloaded function "atomicAdd" matches the argument list argument types are: (double *, double) detected during instantiation of "void VecQuant8MatMulKernel(const scalar_t *, const int *, scalar_t *, const scalar_t *, const scalar_t *, int, int, int, int) [with scalar_t=double]" (359): here

4 errors detected in the compilation of "quant_cuda_kernel.cu". error: command 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\bin\nvcc.exe' failed with exit code 1 Press any key to continue . . .

a seemingly infinitely long error before that part of it

Mar 21 '23 01:03 gsgoldma

Like I said, you are better off installing the wheel file that I uploaded: https://github.com/oobabooga/text-generation-webui/issues/457#issuecomment-1477075495 TBH, I don't really know why compilation started working for me. At first, I was getting constant errors like you. Then my computer restarted overnight and in the morning it just started working. Maybe the restart did it? No clue, because I had tried it before. Just place that wheel file in the gptq folder and install it within the conda environment.

Mar 21 '23 01:03 jllllll

that file causes all sorts of problems. trying WSL now and just FLOODED with errors at every turn

Mar 21 '23 04:03 gsgoldma

The wheel is for Windows, not WSL. It will allow you to skip the GPTQ-for-LLaMa compilation that wasn't working.

Mar 21 '23 08:03 jllllll

Like I said, you are better off installing the wheel file that I uploaded: #457 (comment) TBH, I don't really know why compilation started working for me. At first, I was getting constant errors like you. Then my computer restarted overnight and in the morning it just started working. Maybe the restart did it? No clue, because I had tried it before. Just place that wheel file in the gptq folder and install it within the conda environment.

Similar case to me as well. I installed VS BuildTools but the setup_cuda.py file couldn't find the compiler and if it did after changing the PATH and other parts of the environment, it would give me some access denied error even after I gave it admin privileges. Eventually, after messing around with it enough, it just started working with the original commands. Might have had something to do with restarting Command Prompt or deleting my environment and starting over.

Mar 21 '23 14:03 xNul

This issue has been closed due to inactivity for 30 days. If you believe it is still relevant, please leave a comment below.

Apr 20 '23 23:04 github-actions[bot]

mismatched cuda error

Describe the bug

Is there an existing issue for this?

Reproduction

Screenshot

Logs

System Info

All requested packages already installed.