libcublas.so.12: cannot open shared object file: No such file or directory

[ojipadeson]

My environment is: linux system：Ubuntu 18.04 cuda: 12.0 torch：1.13.1+cu116 bitsandbytes：0.38.1(compile from source)

Then the error is：

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /home/fangrui/miniconda3/envs/ziya/lib/python3.9/site-packages/bitsandbytes-0.38.1-py3.9.egg/bitsandbytes/libbitsandbytes_cuda120.so
/home/fangrui/miniconda3/envs/ziya/lib/python3.9/site-packages/bitsandbytes-0.38.1-py3.9.egg/bitsandbytes/cuda_setup/main.py:145: UserWarning: /home/fangrui/miniconda3/envs/ziya did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
  warn(msg)
/home/fangrui/miniconda3/envs/ziya/lib/python3.9/site-packages/bitsandbytes-0.38.1-py3.9.egg/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/lib')}
  warn(msg)
/home/fangrui/miniconda3/envs/ziya/lib/python3.9/site-packages/bitsandbytes-0.38.1-py3.9.egg/bitsandbytes/cuda_setup/main.py:145: UserWarning: /usr/local/cuda/lib did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
  warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 120
CUDA SETUP: Loading binary /home/fangrui/miniconda3/envs/ziya/lib/python3.9/site-packages/bitsandbytes-0.38.1-py3.9.egg/bitsandbytes/libbitsandbytes_cuda120.so...
libcublas.so.12: cannot open shared object file: No such file or directory
CUDA SETUP: Something unexpected happened. Please compile from source:
git clone [email protected]:TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=120
python setup.py install
Traceback (most recent call last):
  File "/home/fangrui/miniconda3/envs/ziya/lib/python3.9/runpy.py", line 188, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/home/fangrui/miniconda3/envs/ziya/lib/python3.9/runpy.py", line 147, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/home/fangrui/miniconda3/envs/ziya/lib/python3.9/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/home/fangrui/miniconda3/envs/ziya/lib/python3.9/site-packages/bitsandbytes-0.38.1-py3.9.egg/bitsandbytes/__init__.py", line 6, in <module>
    from . import cuda_setup, utils, research
  File "/home/fangrui/miniconda3/envs/ziya/lib/python3.9/site-packages/bitsandbytes-0.38.1-py3.9.egg/bitsandbytes/research/__init__.py", line 1, in <module>
    from . import nn
  File "/home/fangrui/miniconda3/envs/ziya/lib/python3.9/site-packages/bitsandbytes-0.38.1-py3.9.egg/bitsandbytes/research/nn/__init__.py", line 1, in <module>
    from .modules import LinearFP8Mixed, LinearFP8Global
  File "/home/fangrui/miniconda3/envs/ziya/lib/python3.9/site-packages/bitsandbytes-0.38.1-py3.9.egg/bitsandbytes/research/nn/modules.py", line 8, in <module>
    from bitsandbytes.optim import GlobalOptimManager
  File "/home/fangrui/miniconda3/envs/ziya/lib/python3.9/site-packages/bitsandbytes-0.38.1-py3.9.egg/bitsandbytes/optim/__init__.py", line 6, in <module>
    from bitsandbytes.cextension import COMPILED_WITH_CUDA
  File "/home/fangrui/miniconda3/envs/ziya/lib/python3.9/site-packages/bitsandbytes-0.38.1-py3.9.egg/bitsandbytes/cextension.py", line 20, in <module>
    raise RuntimeError('''
RuntimeError: 
        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

How should I solve the issue?

[dt-ahmed-touila]

The torch version (1.13.1+cu116) you are using is not compatible with CUDA 12. Maybe consider upgrading to a cu12 torch version. For now only an experimental version is available => 2.1.0.dev20230607+cu121 but check the install matrix for updates. Also, you might to install bitsandbytes from source

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[ruifengma]

I have pytorch version 2.1.2+cu121 with CUDA version 12.2 and I'm getting the same issue (CUDA can be detected by pytorch)

[deep-pipeline]

@ruifengma just wondering - is this issue still live for you or could it be closed now?

[Titus-von-Koeller]

Thanks @deep-pipeline, good point, closing due to inactivity. Please open a detailed new issue with exact steps to reproduce if anyone encounters similar seeming issues.