UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'AzureExecutionProvider, CPUExecutionProvider'

[PcChip]

It took a bit of effort to get everything running, but I feel like I'm almost there

when starting, it will read "All neural network" and then stop speaking, also audio detection is extremely slow

This is on Kubuntu 23.10 with an RTX4090

I'm seeing a warning about CUDAExecutionProvider not being available, from onnxruntime. Googling that led me to install onnxruntime-gpu via pip, but that didn't help

can anyone provide any assistance?

[dnhkng]

Could it be a CUDA version error? Are you on 11 or 12?

[kaminoer]

Install onnxruntime-gpu.

[PcChip]

Could it be a CUDA version error? Are you on 11 or 12?

nvidia-smi says i'm on 12.4

Install onnxruntime-gpu.

thanks, but as i said in my initial post, I already tried that

[kaminoer]

Oh sorry, missed that part. I was running into the very same warning and installing onnxruntime-gpu fixed the issue for me.

[dnhkng]

There's a big refactor incoming. I hope to have both cleaner code, and installers for Windows and Mac soon after.

[PcChip]

okay, i fixed that error!

pip uninstall onnxruntime  
pip uninstall onnxruntime-gpu  
pip install onnxruntime-gpu --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/

[dnhkng]

Also, IIRC, you have to completely remove the standard onnxruntime before you install the GPU version. There may be other CPU packages you also need to remove before installing the onnxruntime-gpu package.

From the error message, this is surely the issue, as the CUDA version is not being detected.

[PcChip]

so now it speaks the entire first sentence like you would expect, but listening for input seems stalled out and not responsive

also I have a new error:

2024-05-04 13:57:21.911021942 [E:onnxruntime:Default, provider_bridge_ort.cc:1548 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1209 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcudnn.so.8: cannot open shared object file: No such file or directory

[dnhkng]

Looks like the path for cuda is not set up correctly. Check your path, and if it's not there, the cuda documention on adding the cuda library files.

Or maybe you haven't yet installed CUDNN, which is separate to CUDA.

[PcChip]

I ran the following and I think I'm getting closer:
sudo apt install nvidia-cudnn nvidia-cuda-toolkit

now the errors are gone and just replaced with warnings:
2024-05-04 14:10:35.336634256 [W:onnxruntime:, transformer_memcpy.cc:74 ApplyImpl] 28 Memcpy nodes are added to the graph torch_jit for CUDAExecutionProvider. It might have negative impact on performance (including unable to run CUDA graph). Set session_options.log_severity_level=1 to see the detail logs before this message. 2024-05-04 14:10:35.343949704 [W:onnxruntime:, session_state.cc:1166 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf. 2024-05-04 14:10:35.343955406 [W:onnxruntime:, session_state.cc:1168 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.

however speech detection still takes about 10-15 seconds to detect one word from me

[dnhkng]

Not seen that before. Maybe now try uninstalling and reinstalling the onnxruntime-gpu packages. Maybe they can now be configured correctly? Other than that, I'm a bit stumped. Are you in WSL or a native Kubuntu?

[PcChip]

okay... did the following:

added repository to sources list:
https://askubuntu.com/questions/1491254/installing-cuda-on-ubuntu-23-10-libt5info-not-installable#:~:text=The%20libtinfo5%20package,while%20installing%20CUDA

ran the following commands listed here:

https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=22.04&target_type=deb_local

removed and reinstalled onnxruntime-gpu (using special version i posted above)

now the errors are all gone!

it does seem to be working, but there is still a lag between me speaking and it actually detecting what I am saying... hmm

[PcChip]

okay.. rebuilt whisper with CUDA again, then ran WHISPER_CUDA=1 make libwhisper.so , then overwrote the old .so with this new one, and I think it's working now...

although half the time it doesn't read it's output aloud to me... weird

[dnhkng]

Closing this now, as the other error seems unrelated.