CUDA 12.5 support or GPU acceleration not working after graphics driver update

[CodeMazeSolver]

Hey there, I'm running

LocalAI version:

docker run --rm -ti --gpus all -p 8080:8080 -e DEBUG=true -v $PWD/models:/models --name local-ai localai/localai:latest-aio-gpu-nvidia-cuda-12 --models-path /models --context-size 1000 --threads 14

LocalAI version: v2.15.0 (f69de3be0d274a676f1d1cd302dc4699f1b5aaf0)

Environment, CPU architecture, OS, and Version:

13th Gen Intel(R) Core(TM) i9-13900H 2.60 GHz, on Windows 11 with Docker for Windows.

Describe the bug

I get this debug message right before the model is loaded.

stderr ggml_cuda_init: failed to initialize CUDA: named symbol not found

Which indicated to me that the models will not use GPU support. However, this worked just fine before.

After updating the graphics driver, the CUDA version was changed, too. From CUDA version 12.4 to 12.5. It seems like the CUDA environment is no longer used by any LLM. However, the CUDA version is detected correctly when starting the LocalAI Docker container.

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.42.03              Driver Version: 555.85         CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090 ...    On  |   00000000:01:00.0  On |                  N/A |
| N/A   48C    P8              7W /  105W |     148MiB /  16376MiB |      1%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
NVIDIA GPU detected. Attempting to find memory size...
Total GPU Memory: 16376 MiB

Instead of utilizing the GPU, the application uses the fallback and runs only on the CPU.

To Reproduce

Expected behavior

Utilizing the GPU.

Logs

Here are the full logs for the mistral-7b-instruct-v0.1.Q5_K_M.gguf model, but I tried several models that worked before. None utilize the GPU after installing the new graphics driver.

localai.log

Additional context

Also checking in the task manager shows that there is no GPU usage taking place.