Cannot build with CUDA on Win11

[riedgar-ms]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [x] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• [x] I carefully followed the README.md.
• [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• [x] I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

Compilation against CUDA to succeed

Current Behavior

The build fails. In a freshly created Python 3.12 conda environment, I run (in Powershell):

$Env:CMAKE_ARGS = "-DLLAMA_CUDA=on"
pip install -vv --no-cache-dir --force-reinstall llama-cpp-python

and the cmake step fails:

Building wheels for collected packages: llama-cpp-python
  Created temporary directory: C:\Users\riedgar\AppData\Local\Temp\pip-wheel-qsal90j4
  Destination directory: C:\Users\riedgar\AppData\Local\Temp\pip-wheel-qsal90j4
  Running command Building wheel for llama-cpp-python (pyproject.toml)
  *** scikit-build-core 0.8.2 using CMake 3.29.2 (wheel)
  *** Configuring CMake...
  2024-04-17 14:00:05,658 - scikit_build_core - WARNING - Can't find a Python library, got libdir=None, ldlibrary=None, multiarch=None, masd=None
  loading initial cache file C:\Users\riedgar\AppData\Local\Temp\tmpm5zopz_3\build\CMakeInit.txt
  -- Building for: Visual Studio 17 2022
  -- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.22631.
  -- The C compiler identification is MSVC 19.39.33523.0
  -- The CXX compiler identification is MSVC 19.39.33523.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- Found Git: C:/Program Files/Git/cmd/git.exe (found version "2.44.0.windows.1")
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
  -- Looking for pthread_create in pthreads
  -- Looking for pthread_create in pthreads - not found
  -- Looking for pthread_create in pthread
  -- Looking for pthread_create in pthread - not found
  -- Found Threads: TRUE
  -- Found CUDAToolkit: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.4/include (found version "12.4.131")
  -- CUDA found
  CMake Error at C:/Users/riedgar/AppData/Local/Temp/pip-build-env-bzklc6wj/normal/Lib/site-packages/cmake/data/share/cmake-3.29/Modules/CMakeDetermineCompilerId.cmake:563 (message):
    No CUDA toolset found.
  Call Stack (most recent call first):
    C:/Users/riedgar/AppData/Local/Temp/pip-build-env-bzklc6wj/normal/Lib/site-packages/cmake/data/share/cmake-3.29/Modules/CMakeDetermineCompilerId.cmake:8 (CMAKE_DETERMINE_COMPILER_ID_BUILD)
    C:/Users/riedgar/AppData/Local/Temp/pip-build-env-bzklc6wj/normal/Lib/site-packages/cmake/data/share/cmake-3.29/Modules/CMakeDetermineCompilerId.cmake:53 (__determine_compiler_id_test)
    C:/Users/riedgar/AppData/Local/Temp/pip-build-env-bzklc6wj/normal/Lib/site-packages/cmake/data/share/cmake-3.29/Modules/CMakeDetermineCUDACompiler.cmake:131 (CMAKE_DETERMINE_COMPILER_ID)
    vendor/llama.cpp/CMakeLists.txt:387 (enable_language)

I have tried some of the solutions mentioned in #721 but they have not helped.

Environment and Context

• Physical (or virtual) hardware you are using, e.g. for Linux:

Core i7-13800 GeForce 4060 RTX Laptop

• Operating System, e.g. for Linux:

Windows 11

• SDK version, e.g. for Linux:

> python --version
3.12.2
> nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:30:10_Pacific_Daylight_Time_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0
> nvidia-smi
Wed Apr 17 14:35:03 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 551.78                 Driver Version: 551.78         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                     TCC/WDDM  | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4060 ...  WDDM  |   00000000:01:00.0 Off |                  N/A |
| N/A   38C    P8              7W /   14W |       0MiB /   8188MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

Failure Information (for bugs)

See above

Steps to Reproduce

See above

[diakt]

I believe setting an env var for CUDATOOLKITDIR resolved this for me, even though I was seeing Cuda Toolkit found as well... for powershell it was $env:CUDATOOLKITDIR="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v<yourversion>" . Take a stab at that.

[3eeps]

im not sure why it isnt being fixed, but this worked for me: https://github.com/abetlen/llama-cpp-python/discussions/871

set "CMAKE_ARGS=-DLLAMA_CUBLAS=on" && pip install llama-cpp-python

[HansRandomizer]

Sometimes it can help to copy some files from cuda to the MSBuildExtensions folder of your Visual Studio… see https://stackoverflow.com/questions/56636714/cuda-compile-problems-on-windows-cmake-error-no-cuda-toolset-found

worked for me

[riedgar-ms]

@diakt thanks for the suggestion, but I'm not having any luck with that either :-(

@3eeps I was trying the powershell variant on that :-(

[diakt]

@riedgar-ms Take a look at the resolution of #1368. I strongly believe it may help you. I was dealing with a different set of problems later on but I experienced the exact problem you did. A lot of these dependencies are unglaublich messy and I'm not sure my initial recommended fix was even close to the right answer.

[Ray0907]

I encountered the same issue. Here’s my solution: copy the contents from NVIDIA GPU Computing Toolkit\CUDA\v12.1\extras\visual_studio_integration\MSBuildExtensions to Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170. This worked for me.

[javierxio]

The correct answer (or at least what worked for me) is to uninstall the standalone VS Build Tools and from Visual Studio Installer install the Desktop Development with C++.

Also, needed to go through the reinstall of CUDA program. Took me a day to get it working! Hope it helps!

Then, I ran this command:

set "CMAKE_ARGS=-DLLAMA_CUBLAS=on" && python -m pip install llama-cpp-python