TensorFlow 2.10 causes trouble!

[MikePelton]

Hi - many thanks for releasing this code! Tensorflow 2.10 is now (since about September 1st 2022) the default version for a pip install but running in a CUDA 11.2.2 CUDNN 8.1 environment you'll see error messages characterised by "Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered". The solution in the context of this code is to downgrade to TF 2.9.2 - I suggest you change the requirements.txt file to reflect the specific version.

[MikePelton]

I should add this is on Ubuntu 20.04.

[jonbarron]

Thanks for the heads up! Would you mind pushing a CL with that change to requirements.txt?

[MikePelton]

No problem - will pick that up first thing tomorrow.

[MikePelton]

Hmm - this is more complicated than I'd expected! The version of TensorFlow needs to match the versions of the CUDA toolkit and cuDNN, so my using Ubuntu 20.04, CUDA 11.2.2 and cuDNN 8.1 happens to sit nicely with TensorFlow 9.2.2, but later CUDA toolkits will call for later TensorFlow versions (the full story is here: https://www.tensorflow.org/install/source#gpu_support_3) so mandating a version of TensorFlow in requirements.txt might actually be doing a disservice to people building in other environments. Short of providing a dockerfile or Docker image I'm not sure how we'd make this plain sailing!

[MikePelton]

You gotta laugh - at render time after 5000 cycles there's a runtime error saying that version of Tensorflow needs cuDNN 8.2.4. Watch this space :-)

[MikePelton]

Installing cuDNN 8.2.4 seems to fix this - with hindsight doing a "conda install..." of tensorflow would have made more sense as it deals with the dependencies more elegantly than pip.

[jsdevtom]

Hi - many thanks for releasing this code! Tensorflow 2.10 is now (since about September 1st 2022) the default version for a pip install but running in a CUDA 11.2.2 CUDNN 8.1 environment you'll see error messages characterised by "Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered". The solution in the context of this code is to downgrade to TF 2.9.2 - I suggest you change the requirements.txt file to reflect the specific version.

Downgrading tensorflow to 2.9.2 solved this for me. Thanks.

[William-vds]

Hi, I am using Ubuntu 22.04 and have the same error: Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered I have Cuda 11.7 installed. How can I solve this?

[MikePelton]

Hi William - you need to force the version of tensorflow that plays nicely with the version of the CUDA toolkit and CUDNN that you have , which the link above should help with, but we do now know that CUDA 11.2.2, CUDNN 8.1 and TF 2.9.2 will work. You might get away with downgrading tensorflow from 2.10 to 2.9.2 as I did: pip3 install --upgrade tensorflow==2.9.2 ...but that doesn't work I'd recommend downgrading your CUDA and going from there. If you find a combo that works with 11.7, please share!

[fantauzzi]

Same issue here, with Ubuntu 22.04.1. Here the combination that worked on my system:

• tensorflow 2.9.2 (installed with pip)
• cuda 11.5.1 (installed with sudo apt install nvidia-cuda-toolkit)
• cudnn 8.2.4.15 (apparently for cuda 11.4, installed with sudo apt install nvidia-cudnn)
• nvidia-driver-510 (installed with sudo apt install nvidia-driver-510 after installing cuda and cudnn)
• reboot