TensorFlow 2.10 causes trouble!

[MikePelton]

Hi - many thanks for releasing this code! Tensorflow 2.10 is now (since about September 1st 2022) the default version for a pip install but running in a CUDA 11.2.2 CUDNN 8.1 environment you'll see error messages characterised by "Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered". The solution in the context of this code is to downgrade to TF 2.9.2 - I suggest you change the requirements.txt file to reflect the specific version.

[MikePelton]

I should add this is on Ubuntu 20.04.

[jonbarron]

Thanks for the heads up! Would you mind pushing a CL with that change to requirements.txt?

[MikePelton]

No problem - will pick that up first thing tomorrow.

[MikePelton]

Hmm - this is more complicated than I'd expected! The version of TensorFlow needs to match the versions of the CUDA toolkit and cuDNN, so my using Ubuntu 20.04, CUDA 11.2.2 and cuDNN 8.1 happens to sit nicely with TensorFlow 9.2.2, but later CUDA toolkits will call for later TensorFlow versions (the full story is here: https://www.tensorflow.org/install/source#gpu_support_3) so mandating a version of TensorFlow in requirements.txt might actually be doing a disservice to people building in other environments. Short of providing a dockerfile or Docker image I'm not sure how we'd make this plain sailing!

[MikePelton]

You gotta laugh - at render time after 5000 cycles there's a runtime error saying that version of Tensorflow needs cuDNN 8.2.4. Watch this space :-)

[MikePelton]

Installing cuDNN 8.2.4 seems to fix this - with hindsight doing a "conda install..." of tensorflow would have made more sense as it deals with the dependencies more elegantly than pip.

[jsdevtom]

Hi - many thanks for releasing this code! Tensorflow 2.10 is now (since about September 1st 2022) the default version for a pip install but running in a CUDA 11.2.2 CUDNN 8.1 environment you'll see error messages characterised by "Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered". The solution in the context of this code is to downgrade to TF 2.9.2 - I suggest you change the requirements.txt file to reflect the specific version.

Downgrading tensorflow to 2.9.2 solved this for me. Thanks.

[William-vds]

Hi, I am using Ubuntu 22.04 and have the same error: Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered I have Cuda 11.7 installed. How can I solve this?

[MikePelton]

Hi William - you need to force the version of tensorflow that plays nicely with the version of the CUDA toolkit and CUDNN that you have , which the link above should help with, but we do now know that CUDA 11.2.2, CUDNN 8.1 and TF 2.9.2 will work. You might get away with downgrading tensorflow from 2.10 to 2.9.2 as I did: pip3 install --upgrade tensorflow==2.9.2 ...but that doesn't work I'd recommend downgrading your CUDA and going from there. If you find a combo that works with 11.7, please share!

[fantauzzi]

Same issue here, with Ubuntu 22.04.1. Here the combination that worked on my system:

• tensorflow 2.9.2 (installed with pip)
• cuda 11.5.1 (installed with sudo apt install nvidia-cuda-toolkit)
• cudnn 8.2.4.15 (apparently for cuda 11.4, installed with sudo apt install nvidia-cudnn)
• nvidia-driver-510 (installed with sudo apt install nvidia-driver-510 after installing cuda and cudnn)
• reboot

[MaxBDSGH]

Hi MikePelton, your suggestions did work in my environment too. Thank you.

I created a new environment in Anaconda with Python 3.8. Before installing tensorflow package I installed following packages:

• nvidia-pyindex (pip install nvidia-pyindex executed in the new environment);
• nvidia-tensorrt 7.2.3.4 (pip install nvidia-tensorrt==7.2.3.4 executed in the new environment);
• cudatoolkit 11.2 and cudnn 8.1.0 (conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0 executed in the new environment);
• tensorflow 2.9.2 (pip install tensorflow==2.9.2 executed in the new environment).

With tensorflow 2.10.0 I was getting the error message: Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered.

Ciao

[fantauzzi]

Hi MikePelton, your suggestions did work in my environment too. Thank you. [...] @MaxBDSGH may I ask what OS/version are you on?

[MaxBDSGH]

Hi fantauzzi, I'm on Ubuntu 22.04.1 LTS.

Hi MikePelton, your suggestions did work in my environment too. Thank you. [...] @MaxBDSGH may I ask what OS/version are you on?

[mauriciocramos]

Installing cuDNN 8.2.4 seems to fix this - with hindsight doing a "conda install..." of tensorflow would have made more sense as it deals with the dependencies more elegantly than pip.

You @MikePelton was very elegant by saying "elegantly" :-) Some beliefs I'd like to share:

1. conda considers pip installations as "read-only", just recommends to install them at last and even export them (yaml)
2. pip otherwise ignores conda installs which I might lead to redundant if not broken/impossible complete installations, forcing users to create separate environments.
3. Almost all mainstream frameworks employs conda packages.
4. Although I'm a DS/ML/DL researcher coding Python, as many, I consider my self a "user" which needs to quickly and easily setup mixed frameworks, until the day I will be able to use only one, e.g. RAPIDS.
5. By the way RAPIDS which has to do with nvidia, uses conda, as numba, pytorch, spacy and sklearn do. I hope one day Tensorflow will officially support conda and and nvidia would do the same to things like tensorrt.

[Romeo-CC]

I met this problem in tensorflow2.14.0 Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered

CUDA_VERSION: 11.8 CUDNN_VERSION: 8.9.4 OS: ubuntu 23.04

[netl0tuz]

I met this problem in tensorflow2.14.0 Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered

CUDA_VERSION: 11.8 CUDNN_VERSION: 8.9.4 OS: ubuntu 23.04

use version 2.13 instead , i use tensorflow:2.13.0-gpu container from docker hub and it works.( on 2.14 i have same problem)