System Info

root@eedf435b757f:/opt/pytorch/lightning-thunder# uname -a Linux eedf435b757f 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

GPU - A100

bitsandbytes version 0.42.0

Reproduction

Tried to set BNB_CUDA_VERSION=124 in a CUDA 12.6 container, and followed https://github.com/bitsandbytes-foundation/bitsandbytes/blob/main/docs/source/installation.mdx#pytorch-cuda-versions.

got the below error

===================================BUG REPORT===================================

The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/compat/lib'), PosixPath('/usr/local/nvidia/lib'), PosixPath('/usr/local/nvidia/lib64')} The following directories listed in your path were found to be non-existent: {PosixPath('test_quantization_on_meta (call)')} CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... DEBUG: Possible options found for libcudart.so: {PosixPath('/usr/local/cuda/lib64/libcudart.so')} CUDA SETUP: PyTorch settings found: CUDA_VERSION=126, Highest Compute Capability: 8.0. CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md CUDA SETUP: Required library version not found: libbitsandbytes_cuda126.so. Maybe you need to compile it from source? CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...

================================================ERROR===================================== CUDA SETUP: CUDA detection failed! Possible reasons:

You need to manually override the PyTorch CUDA version. Please see: "https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA driver not installed
CUDA not installed
You have multiple conflicting CUDA libraries
Required library not pre-compiled for this bitsandbytes release! CUDA SETUP: If you compiled from source, try again with make CUDA_VERSION=DETECTED_CUDA_VERSION for example, make CUDA_VERSION=113. CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version via conda list | grep cuda. ================================================================================

CUDA SETUP: Something unexpected happened. Please compile from source: git clone https://github.com/TimDettmers/bitsandbytes.git cd bitsandbytes CUDA_VERSION=126 python setup.py install CUDA SETUP: Setup Failed!

Expected behavior

Should be able to pick up CUDA 12.4

Aug 13 '24 02:08 tinglvv

I met the same error. GPU:RTX 4090D Python ver:3.11.5 torch ver:2.4.0+cu124 bitsandbytes ver 0.43.2

+-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 2236 C+G C:\Windows\System32\LogonUI.exe N/A | +-----------------------------------------------------------------------------------------+`

(venv) PS D:\AI_draw\sd-scripts> nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2024 NVIDIA Corporation Built on Tue_Feb_27_16:28:36_Pacific_Standard_Time_2024 Cuda compilation tools, release 12.4, V12.4.99 Build cuda_12.4.r12.4/compiler.33961263_0

(venv) PS D:\AI_draw\sd-scripts> python -m bitsandbytes False

===================================BUG REPORT=================================== D:\AI_draw\sd-scripts\venv\Lib\site-packages\bitsandbytes\cuda_setup\main.py:183: UserWarning: Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

warn(msg)

The following directories listed in your path were found to be non-existent: {WindowsPath('/Users/Administrator')} The following directories listed in your path were found to be non-existent: {WindowsPath('/DESKTOP-K5DEQA5')} The following directories listed in your path were found to be non-existent: {WindowsPath('C:/Users/Administrator/Documents/WindowsPowerShell/Modules')} DEBUG: Possible options found for libcudart.so: {WindowsPath('C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.4/bin/cudart64_12.dll')} CUDA SETUP: PyTorch settings found: CUDA_VERSION=124, Highest Compute Capability: 8.9. CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md CUDA SETUP: Required library version not found: libbitsandbytes_cuda124.dll. Maybe you need to compile it from source? CUDA SETUP: Defaulting to libbitsandbytes_cpu.dll...

================================================ERROR===================================== CUDA SETUP: CUDA detection failed! Possible reasons:

You need to manually override the PyTorch CUDA version. Please see: "https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA driver not installed
CUDA not installed
You have multiple conflicting CUDA libraries
Required library not pre-compiled for this bitsandbytes release! CUDA SETUP: If you compiled from source, try again with make CUDA_VERSION=DETECTED_CUDA_VERSION for example, make CUDA_VERSION=118. CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version via conda list | grep cuda. ================================================================================

CUDA SETUP: Something unexpected happened. Please compile from source: git clone https://github.com/TimDettmers/bitsandbytes.git cd bitsandbytes CUDA_VERSION=124 python setup.py install CUDA SETUP: Setup Failed! Traceback (most recent call last): File "", line 189, in _run_module_as_main File "", line 148, in get_module_details File "", line 112, in get_module_details File "D:\AI_draw\sd-scripts\venv\Lib\site-packages\bitsandbytes_init.py", line 6, in from . import cuda_setup, research, utils File "D:\AI_draw\sd-scripts\venv\Lib\site-packages\bitsandbytes\research_init.py", line 2, in from .autograd._functions import ( File "D:\AI_draw\sd-scripts\venv\Lib\site-packages\bitsandbytes\research\autograd_functions.py", line 8, in from bitsandbytes.autograd.functions import GlobalOutlierPooler, MatmulLtState File "D:\AI_draw\sd-scripts\venv\Lib\site-packages\bitsandbytes\autograd_init.py", line 1, in from ._functions import get_inverse_transform_indices, undo_layout File "D:\AI_draw\sd-scripts\venv\Lib\site-packages\bitsandbytes\autograd_functions.py", line 10, in import bitsandbytes.functional as F File "D:\AI_draw\sd-scripts\venv\Lib\site-packages\bitsandbytes\functional.py", line 17, in from .cextension import COMPILED_WITH_CUDA, lib File "D:\AI_draw\sd-scripts\venv\Lib\site-packages\bitsandbytes\cextension.py", line 17, in raise RuntimeError(''' RuntimeError: CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

Aug 13 '24 05:08 Wenaka2004

I solved it by pip list check all intalled versions of bitsandbytes and pip uninstall bitsandbytes to uninstall all of them then pip install --upgrade "bitsandbytes>=0.43.2"

Aug 13 '24 06:08 Wenaka2004

Thanks @Wenaka2004! for me I got this error after upgrading to 0.43.3

self = <CDLL '/usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cpu.so', handle 5566bc1f32e0 at 0x7f635045e680>
name_or_ordinal = 'cquantize_blockwise_bf16_nf4'

    def __getitem__(self, name_or_ordinal):
>       func = self._FuncPtr((name_or_ordinal, self))
E       AttributeError: /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cquantize_blockwise_bf16_nf4

/usr/lib/python3.10/ctypes/__init__.py:392: AttributeError
---------------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------------
WARNING  bitsandbytes.cextension:cextension.py:94 Could not find the bitsandbytes CUDA binary at PosixPath('/usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda126.so')
WARNING  bitsandbytes.cextension:cextension.py:101 The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.

Aug 13 '24 21:08 tinglvv

There's two different issues here.

@tinglvv We're not yet distributing a CUDA 12.6 binary, but it looks like your environment is still trying to load for that version. I would double check how you set BNB_CUDA_VERSION=124 - if this was visible it would instead load libbitsandbytes_cuda124.so.

@Wenaka2004 In your situation, we only shipped CUDA 12.4 binaries for Windows starting with 0.43.3, so this is why upgrading resolved that issue.

Aug 14 '24 14:08 matthewdouglas

BNB_CUDA_VERSION=124

can you tell when cuda 126 will be out for your plugin?

Dec 01 '24 14:12 flashbackpoductions

@flashbackpoductions We will have a 12.6 binary included in our next release to PyPI which will be very soon.

Dec 02 '24 14:12 matthewdouglas

It's Jan 30th 2025....still waiting>

Could not find the bitsandbytes CUDA binary at WindowsPath('C:/Users/xxxxx/source/repos/AI/runtimes/bin/windows/Python312/Lib/site-packages/bitsandbytes/libbitsandbytes_cuda126.dll') The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.

Jan 30 '25 17:01 ukaprch

我通过 pip list 检查所有已安装的 bitsandbytes 版本并 pip uninstall bitsandbytes 卸载它们来解决这个问题 pip install --upgrade "bitsandbytes>=0.43.2" 我也是用博主的这个方法解决的，感谢！

Feb 20 '25 12:02 huhuba

Thanks for the update.

Feb 22 '25 19:02 ukaprch

We've added the CUDA 12.6 builds in the v0.45.0 release. Please upgrade and try if it resolves the issue. If it persists please create a new issue so we can review.

Feb 28 '25 14:02 matthewdouglas

I solved it by pip list check all intalled versions of bitsandbytes and pip uninstall bitsandbytes to uninstall all of them then pip install --upgrade "bitsandbytes>=0.43.2"

i solved it follow this answer

Jul 10 '25 06:07 myscius

Unable to override PyTorch CUDA Version

System Info

Reproduction

===================================BUG REPORT===================================

Expected behavior

warn(msg)