libcudnn error while running on Google colab with GPU

[soham583]

I am not able to run extended version on google colab with GPU. with cpu it is working but when used with GPU, when it starts with whisper it is showing

'Unable to load any of {libcudnn_cnn.so.9.1.0, libcudnn_cnn.so.9.1, libcudnn_cnn.so.9, libcudnn_cnn.so} Invalid handle. Cannot load symbol cudnnCreateConvolutionDescriptor'

and stops.

[mmccollow]

I'm having the same problem on my local PC running Pop!_OS 22.04 LTS.

[ryan-steed-usa]

Hello,

I'm not too familiar with Colab (since I don't use it) but if you can locate the path to these libcudnn files, you can extend the LD_LIBRARY_PATH variable to point to it. googlecolab/colabtools#1317 shows an example.

According to googlecolab/colabtools#2595, the path might be /usr/local/lib. If so, the following might work per: https://github.com/googlecolab/colabtools/issues/1317#issuecomment-747586346

os.environ["LD_LIBRARY_PATH"] = f"/usr/local/lib:{os.environ.get('LD_LIBRARY_PATH', '')}"

To fix the issue within a Linux Podman container, I've used a command like the following to find the path:

find / -iname 'libcudnn*'

Which ended up as $HOME/.local/lib/python3.10/site-packages/nvidia/cudnn/lib/. And the fix in this case could be:

export LD_LIBRARY_PATH=$HOME/.local/lib/python3.10/site-packages/nvidia/cudnn/lib:$HOME/.local/lib/python3.10/site-packages/nvidia/cuda_runtime/lib:$LD_LIBRARY_PATH

Good luck!

Edit: fix path for hypothetical Linux example