whisper.cpp
whisper.cpp copied to clipboard
ggerganov
Request for Guidance on Running whisper.cpp with cuBLAS on Windows
I am new to both Whisper.cpp and C++, and I would appreciate some guidance on how to run whisper.cpp with cuBLAS. I have successfully downloaded the Windows binaries (whisper-blas-bin-x64.zip) and executed main.exe using the following command:
$ ./main.exe -m ./model/ggml-large.bin -l cs -osrt audio_16khz.wav
My system has a working CUDAToolkit installed, as I can use it with PyTorch (torch.cuda.is_available() >>> true). The CUDA Toolkit is installed in the following directory: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1.
I expect the find_package(CUDAToolkit) function to locate the Toolkit and set BLAS = 0. However, I am unsure if this is happening correctly. Please forgive me if I am overlooking something obvious. Here is the console output I get when running the command:
whisper_init_from_file_no_state: loading model from './model/ggml-large.bin'
whisper_model_load: loading model
...
system_info: n_threads = 4 / 20 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | COREML = 0 |
main: processing 'audio_16khz.wav' (5341101 samples, 333.8 sec), 4 threads, 1 processors, lang = cs, task = transcribe, timestamps = 1 ...
I believe you either need to compile it yourself or they've just added GitHub action for a cublas build so you can get the latest build here: https://github.com/ggerganov/whisper.cpp/actions/runs/4885055280
I already downloaded generated binaries with cuBLAS support. But it is not using BLAS (BLAS = 0) while running.
I already downloaded generated binaries with cuBLAS support. But it is not using BLAS (BLAS = 0 while running.
I have noticed that Windows blas-enabled build of v1.4.0 reports BLAS = 0 as well.
@ggerganov , is it a regression?
@ggerganov , is it a regression?
Looks like it - I can't test on Windows. Hopefully someone will investigate and help resolve
I've built it locally with WHISPER_CUBLAS set and get
System info: AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | COREML = 0 |
system_info: n_threads = 4 / 20 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | COREML = 0 |
Although BLAS = 1， I see in task manager that GPU(RTX 3070TI) is not working, CPU is working.
Compiled locally, works fine: https://github.com/ggerganov/whisper.cpp/issues/89#issuecomment-1574852496
@cddqssc @bbscout Here is a very simple guidance to help you build whisper.cpp with CuBLAS on windows:
Preparation
- Install Visual Studio 2022
- Install latest version CUDA Toolkit
- Download CMake (Portable Windows x64 ZIP)
- Download whisper.cpp source code from GitHub
Build
- Open CMD console
- Input
cmake.exe -S whisper.cpp_source_code_location -B whisper_build_location -D WHISPER_CUBLAS=1 - Input
cmake.exe --build whisper_build_location --config release
Well done!!!
- All binaries can be found in
whisper_build_location\bin\Release
@cddqssc @bbscout Here is a very simple guidance to help you build whisper.cpp with CuBLAS on windows:
Preparation
- Install Visual Studio 2022
- Install latest version CUDA Toolkit
- Download CMake (Portable Windows x64 ZIP)
- Download whisper.cpp source code from GitHub
Build
- Open CMD console
- Input
cmake.exe -S whisper.cpp_source_code_location -B whisper_build_location -D WHISPER_OPENBLAS=1- Input
cmake.exe --build whisper_build_location --config releaseWell done!!!
- All binaries can be found in
whisper_build_location\bin\Release
I get this error
Could NOT find BLAS (missing: BLAS_LIBRARIES)
@cddqssc @bbscout Here is a very simple guidance to help you build whisper.cpp with CuBLAS on windows:
Preparation
- Install Visual Studio 2022
- Install latest version CUDA Toolkit
- Download CMake (Portable Windows x64 ZIP)
- Download whisper.cpp source code from GitHub
Build
- Open CMD console
- Input
cmake.exe -S whisper.cpp_source_code_location -B whisper_build_location -D WHISPER_OPENBLAS=1- Input
cmake.exe --build whisper_build_location --config releaseWell done!!!
- All binaries can be found in
whisper_build_location\bin\ReleaseI get this error Could NOT find BLAS (missing: BLAS_LIBRARIES)
Oops, seems like I goofed up there. The command should be -D WHISPER_CUBLAS=1, not -D WHISPER_OPENBLAS=1. Thanks for bearing with me. Oh, and by the way, if you're looking to compile the OpenBLAS version, you'll need to download and install OpenBLAS first.
@cddqssc @bbscout Here is a very simple guidance to help you build whisper.cpp with CuBLAS on windows:
Preparation
- Install Visual Studio 2022
- Install latest version CUDA Toolkit
- Download CMake (Portable Windows x64 ZIP)
- Download whisper.cpp source code from GitHub
Build
- Open CMD console
- Input
cmake.exe -S whisper.cpp_source_code_location -B whisper_build_location -D WHISPER_CUBLAS=1- Input
cmake.exe --build whisper_build_location --config releaseWell done!!!
- All binaries can be found in
whisper_build_location\bin\Release
Sorry to bother you, sir, I meet some problems when I follow your command
For anyone coming across this in 2024: WHISPER_CUBLAS is deprecated, you have to use GGML_CUDA instead. If you keep getting the deprecation message, delete your build location first. So in order to build directly from the checked-out repository into a /dist folder, you'd use the following:
- Input
cmake.exe -S ./ -B ./dist -D GGML_CUDA=1 - Input
cmake.exe --build ./dist --config release
It might have to do with me using only the Build Tools instead of the full VS installation, but I also had to copy the files from
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\extras\visual_studio_integration\MSBuildExtensions
to
C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\BuildCustomizations
for the build configuration to run through.
Also make sure that your CUDA_PATH environment variable is set correctly.
