ERROR: CUDA kernel vec_dot_q5_0_q8_1_impl has no device code compatible with CUDA arch 520. ggml-cuda.cu was compiled for: 520

[Yasand123]

trafficstars

When running whisper after compiling with WHISPER_CUDA=1 I get this error. It works fine on the CPU without that env variable. I used medium-q5_0 that I downloaded using the script inside models directory.

The command I used ./main -m /home/john/whisper.cpp/models/ggml-medium-q5_0.bin -f samples/jfk.wav

The output (Errors only when compiled with cuda):

isper_init_from_file_with_params_no_state: loading model from '/home/john/whisper.cpp/models/ggml-medium-q5_0.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab       = 51865
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 1024
whisper_model_load: n_audio_head  = 16
whisper_model_load: n_audio_layer = 24
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 1024
whisper_model_load: n_text_head   = 16
whisper_model_load: n_text_layer  = 24
whisper_model_load: n_mels        = 80
whisper_model_load: ftype         = 8
whisper_model_load: qntvr         = 1
whisper_model_load: type          = 4 (medium)
whisper_model_load: adding 1608 extra tokens
whisper_model_load: n_langs       = 99
whisper_backend_init: using CUDA backend
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:   no
ggml_cuda_init: CUDA_USE_TENSOR_CORES: yes
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2060 SUPER, compute capability 7.5, VMM: yes
whisper_model_load:    CUDA0 total size =   538.59 MB
whisper_model_load: model size    =  538.59 MB
whisper_backend_init: using CUDA backend
whisper_init_state: kv self size  =  132.12 MB
whisper_init_state: kv cross size =  147.46 MB
whisper_init_state: compute buffer (conv)   =   28.68 MB
whisper_init_state: compute buffer (encode) =  594.22 MB
whisper_init_state: compute buffer (cross)  =    7.85 MB
whisper_init_state: compute buffer (decode) =  138.87 MB

system_info: n_threads = 4 / 16 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | METAL = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | CUDA = 1 | COREML = 0 | OPENVINO = 0

main: processing 'samples/jfk.wav' (176000 samples, 11.0 sec), 4 threads, 1 processors, 5 beams + best of 5, lang = en, task = transcribe, timestamps = 1 ...

ggml-cuda/vecdotq.cuh:131: ERROR: CUDA kernel vec_dot_q5_0_q8_1_impl has no device code compatible with CUDA arch 520. ggml-cuda.cu was compiled for: 520
CUDA error: the function failed to launch on the GPU
ggml-cuda/vecdotq.cuh:131: ERROR: CUDA kernel vec_dot_q5_0_q8_1_impl has no device code compatible with CUDA arch 520. ggml-cuda.cu was compiled for: 520
  current device: 0, in function ggml_cuda_mul_mat_batched_cublas at ggml-cuda.cu:1848
ggml-cuda/vecdotq.cuh:131: ERROR: CUDA kernel vec_dot_q5_0_q8_1_impl has no device code compatible with CUDA arch 520. ggml-cuda.cu was compiled for: 520
  cublasGemmBatchedEx(ctx.cublas_handle(), CUBLAS_OP_T, CUBLAS_OP_N, ne01, ne11, ne10, alpha, (const void **) (ptrs_src.get() + 0*ne23), CUDA_R_16F, nb01/nb00, (const void **) (ptrs_src.get() + 1*ne23), CUDA_R_16F, nb11/nb10, beta, ( void **) (ptrs_dst.getggml-cuda/vecdotq.cuh:131: ERROR: CUDA kernel vec_dot_q5_0_q8_1_impl has no device code compatible with CUDA arch 520. ggml-cuda.cu was compiled for: 520
() + 0*ne23), cu_data_type, ne01, ne23, cu_compute_type, CUBLAS_GEMM_DEFAULT_TENSOR_OP)
ggml-cuda/vecdotq.cuh:131: ERROR: CUDA kernel vec_dot_q5_0_q8_1_impl has no device code compatible with CUDA arch 520. ggml-cuda.cu was compiled for: 520
GGML_ASSERT: ggml-cuda.cu:60: !"CUDA error"
ggml-cuda/vecdotq.cuh:131: ERROR: CUDA kernel vec_dot_q5_0_q8_1_impl has no device code compatible with CUDA arch 520. ggml-cuda.cu was compiled for: 520
ptrace: Operation not permitted.
No stack.
The program is not being run.
Aborted (core dumped)

SPECs

• Arch Linux 6.8.4-arch1-1
• NVIDIA 2060 super
• 3700x

Software

• cuda 12.4.1-1
• nvidia 550.67-1

I compiled it directly after cloning the repo.