The detected CUDA version (12.1) mismatches the version that was used to compile PyTorch (11.7). Please make sure to use the same CUDA versions.

[Yersi88]

Describe the bug

Following error presents itself when trying to configure for running LLama.

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

Following this tutorial: https://github.com/oobabooga/text-generation-webui/wiki/LLaMA-model

On this step: python setup_cuda.py install

I get this error: RuntimeError: The detected CUDA version (12.1) mismatches the version that was used to compile PyTorch (11.7). Please make sure to use the same CUDA versions

Screenshot

No response

Logs

$ python setup_cuda.py install
running install
/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
running egg_info
writing quant_cuda.egg-info/PKG-INFO
writing dependency_links to quant_cuda.egg-info/dependency_links.txt
writing top-level names to quant_cuda.egg-info/top_level.txt
reading manifest file 'quant_cuda.egg-info/SOURCES.txt'
writing manifest file 'quant_cuda.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
Traceback (most recent call last):
  File "/home/marius/text-generation-webui/repositories/GPTQ-for-LLaMa/setup_cuda.py", line 4, in <module>
    setup(
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
    super().run_command(command)
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/command/install.py", line 74, in run
    self.do_egg_install()
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/command/install.py", line 123, in do_egg_install
    self.run_command('bdist_egg')
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
    super().run_command(command)
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/command/bdist_egg.py", line 165, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/command/bdist_egg.py", line 151, in call_command
    self.run_command(cmdname)
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
    super().run_command(command)
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/command/install_lib.py", line 112, in build
    self.run_command('build_ext')
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
    super().run_command(command)
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 84, in run
    _build_ext.run(self)
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
    self.build_extensions()
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 499, in build_extensions
    _check_cuda_version(compiler_name, compiler_version)
  File "/home/marius/miniconda3/envs/textgen/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 387, in _check_cuda_version
    raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
RuntimeError: 
The detected CUDA version (12.1) mismatches the version that was used to compile
PyTorch (11.7). Please make sure to use the same CUDA versions.

(textgen)

System Info

$ neofetch
                   -`                    marius@archlinux 
                  .o+`                   ---------------- 
                 `ooo/                   OS: Arch Linux x86_64 
                `+oooo:                  Host: X670 GAMING X AX -CF 
               `+oooooo:                 Kernel: 6.2.10-zen1-1-zen 
               -+oooooo+:                Uptime: 57 mins 
             `/:-:++oooo+:               Packages: 1635 (pacman), 20 (flatpak), 7 (snap) 
            `/++++/+++++++:              Shell: zsh 5.9 
           `/++++++++++++++:             Resolution: 2560x1440 
          `/+++ooooooooooooo/`           DE: Plasma 5.27.4 
         ./ooosssso++osssssso+`          WM: kwin 
        .oossssso-`/ossssss+`         WM Theme: Endless 
       -osssssso.      :ssssssso.        Theme: [Plasma], Breeze [GTK3] 
      :osssssss/        osssso+++.       Icons: [Plasma], Relax-Dark-Icons [GTK2/3] 
     /ossssssss/        +ssssooo/-       Terminal: terminator 
   `/ossssso+/:-        -:/+osssso+-     CPU: AMD Ryzen 9 7900X (24) @ 4.700GHz 
  `+sso+:-`                 `.-/+oso:    GPU: AMD ATI 16:00.0 Raphael 
 `++:.                           `-/+/   GPU: NVIDIA GeForce RTX 2080 Ti Rev. A 
 .`                                 `/   Memory: 20931MiB / 63439MiB

[kodicw]

I'm using Fedora 37 and had the same issue. I installed cuda-11-7.x86_64 using the command below

sudo dnf install cuda-11-7.x86_64

then to switch between versions I used this command

sudo update-alternatives --config cuda 

There are 2 programs which provide 'cuda'.

  Selection    Command
-----------------------------------------------
*  1           /usr/local/cuda-12.1
 + 2           /usr/local/cuda-11.7

I hit 2 & Enter then it worked

[renatobellotti]

I'm using Fedora 37 and had the same issue. I installed cuda-11-7.x86_64 using the command below

sudo dnf install cuda-11-7.x86_64

then to switch between versions I used this command

sudo update-alternatives --config cuda 

There are 2 programs which provide 'cuda'.

  Selection    Command
-----------------------------------------------
*  1           /usr/local/cuda-12.1
 + 2           /usr/local/cuda-11.7

I hit 2 & Enter then it worked

Thanks for this hint! I'm also on Fedora 37, but the command sudo dnf install cuda-11-7.x86_64 does not find the package... Which repos did you enable?

[github-actions[bot]]

This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.