Installation succeed with CUDA 12.3, but libcudart.so is not found

[IamGianluca]

System Info

Hi,

I'm running bitsandbytes from a Docker container based on the nvcr.io/nvidia/pytorch:23.12-py3 Docker image. I've installed the library from source, with the following commands:

CUDA_VERSION=123 make cuda12x
python setup.py install

The installation completes successfully, and I can import bitsandbytes from the Python interpreter.

However, executing the following code snippet throws the following error.

from transformers import AutoTokenizer, AutoModelForCausalLM
model = "meta-llama/Llama-2-7b-chat-hf"
tokenizer = AutoTokenizer.from_pretrained(model)
model = AutoModelForCausalLM.from_pretrained(model, device_map="auto", load_in_4bit=True)

False

===================================BUG REPORT===================================
================================================================================
The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/compat/lib'), PosixPath('/usr/local/nvidia/lib'), PosixPath('/usr/local/nvidia/lib64')}
The following directories listed in your path were found to be non-existent: {PosixPath('//matplotlib_inline.backend_inline'), PosixPath('module')}
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
DEBUG: Possible options found for libcudart.so: {PosixPath('/usr/local/cuda/lib64/libcudart.so')}
CUDA SETUP: PyTorch settings found: CUDA_VERSION=123, Highest Compute Capability: 8.6.
CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
CUDA SETUP: Required library version not found: libbitsandbytes_cuda123.so. Maybe you need to compile it from source?
CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...

================================================ERROR=====================================
CUDA SETUP: CUDA detection failed! Possible reasons:
1. You need to manually override the PyTorch CUDA version. Please see: "https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
2. CUDA driver not installed
3. CUDA not installed
4. You have multiple conflicting CUDA libraries
5. Required library not pre-compiled for this bitsandbytes release!
CUDA SETUP: If you compiled from source, try again with `make CUDA_VERSION=DETECTED_CUDA_VERSION` for example, `make CUDA_VERSION=113`.
CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version via `conda list | grep cuda`.
================================================================================

CUDA SETUP: Something unexpected happened. Please compile from source:
git clone https://github.com/TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=123
python setup.py install
CUDA SETUP: Setup Failed!
/usr/local/lib/python3.10/dist-packages/bitsandbytes-0.41.3.post1-py3.10.egg/bitsandbytes/cuda_setup/main.py:167: UserWarning: Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes


/usr/local/lib/python3.10/dist-packages/bitsandbytes-0.41.3.post1-py3.10.egg/bitsandbytes/cuda_setup/main.py:167: UserWarning: /usr/local/lib/python3.10/dist-packages/torch/lib:/usr/local/lib/python3.10/dist-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
File /usr/local/lib/python3.10/dist-packages/transformers/utils/import_utils.py:1382, in _LazyModule._get_module(self, module_name)
   1381 try:
-> 1382     return importlib.import_module("." + module_name, self.__name__)
   1383 except Exception as e:

File /usr/lib/python3.10/importlib/__init__.py:126, in import_module(name, package)
    125         level += 1
--> 126 return _bootstrap._gcd_import(name[level:], package, level)

File <frozen importlib._bootstrap>:1050, in _gcd_import(name, package, level)

File <frozen importlib._bootstrap>:1027, in _find_and_load(name, import_)

File <frozen importlib._bootstrap>:1006, in _find_and_load_unlocked(name, import_)

File <frozen importlib._bootstrap>:688, in _load_unlocked(spec)

File <frozen importlib._bootstrap_external>:883, in exec_module(self, module)

File <frozen importlib._bootstrap>:241, in _call_with_frames_removed(f, *args, **kwds)

File /usr/local/lib/python3.10/dist-packages/transformers/integrations/bitsandbytes.py:11
     10 if is_bitsandbytes_available():
---> 11     import bitsandbytes as bnb
     12     import torch

File /usr/local/lib/python3.10/dist-packages/bitsandbytes-0.41.3.post1-py3.10.egg/bitsandbytes/__init__.py:6
      1 # Copyright (c) Facebook, Inc. and its affiliates.
      2 #
      3 # This source code is licensed under the MIT license found in the
      4 # LICENSE file in the root directory of this source tree.
----> 6 from . import cuda_setup, utils, research
      7 from .autograd._functions import (
      8     MatmulLtState,
      9     bmm_cublas,
   (...)
     13     matmul_4bit
     14 )

File /usr/local/lib/python3.10/dist-packages/bitsandbytes-0.41.3.post1-py3.10.egg/bitsandbytes/research/__init__.py:1
----> 1 from . import nn
      2 from .autograd._functions import (
      3     switchback_bnb,
      4     matmul_fp8_global,
      5     matmul_fp8_mixed,
      6 )

File /usr/local/lib/python3.10/dist-packages/bitsandbytes-0.41.3.post1-py3.10.egg/bitsandbytes/research/nn/__init__.py:1
----> 1 from .modules import LinearFP8Mixed, LinearFP8Global

File /usr/local/lib/python3.10/dist-packages/bitsandbytes-0.41.3.post1-py3.10.egg/bitsandbytes/research/nn/modules.py:8
      7 import bitsandbytes as bnb
----> 8 from bitsandbytes.optim import GlobalOptimManager
      9 from bitsandbytes.utils import OutlierTracer, find_outlier_dims

File /usr/local/lib/python3.10/dist-packages/bitsandbytes-0.41.3.post1-py3.10.egg/bitsandbytes/optim/__init__.py:6
      1 # Copyright (c) Facebook, Inc. and its affiliates.
      2 #
      3 # This source code is licensed under the MIT license found in the
      4 # LICENSE file in the root directory of this source tree.
----> 6 from bitsandbytes.cextension import COMPILED_WITH_CUDA
      8 from .adagrad import Adagrad, Adagrad8bit, Adagrad32bit

File /usr/local/lib/python3.10/dist-packages/bitsandbytes-0.41.3.post1-py3.10.egg/bitsandbytes/cextension.py:20
     19     CUDASetup.get_instance().print_log_stack()
---> 20     raise RuntimeError('''
     21     CUDA Setup failed despite GPU being available. Please run the following command to get more information:
     22 
     23     python -m bitsandbytes
     24 
     25     Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
     26     to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
     27     and open an issue at: [https://github.com/TimDettmers/bitsandbytes/issues](https://github.com/TimDettmers/bitsandbytes/issues%3C/span%3E%3Cspan) style="color:rgb(175,0,0)">''')
     28 lib.cadam32bit_grad_fp32 # runs on an error if the library could not be found -> COMPILED_WITH_CUDA=False

RuntimeError: 
        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
Cell In[4], line 3
      1 model = "meta-llama/Llama-2-7b-chat-hf"
      2 tokenizer = AutoTokenizer.from_pretrained(model)
----> 3 model = AutoModelForCausalLM.from_pretrained(model, device_map="auto", load_in_4bit=True)

File /usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py:566, in _BaseAutoModelClass.from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    564 elif type(config) in cls._model_mapping.keys():
    565     model_class = _get_model_class(config, cls._model_mapping)
--> 566     return model_class.from_pretrained(
    567         pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs
    568     )
    569 raise ValueError(
    570     f"Unrecognized configuration class {config.__class__} for this kind of AutoModel: {cls.__name__}.\n"
    571     f"Model type should be one of {', '.join(c.__name__ for c in cls._model_mapping.keys())}."
    572 )

File /usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py:3476, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, config, cache_dir, ignore_mismatched_sizes, force_download, local_files_only, token, revision, use_safetensors, *model_args, **kwargs)
   3473     keep_in_fp32_modules = []
   3475 if load_in_8bit or load_in_4bit:
-> 3476     from .integrations import get_keys_to_not_convert, replace_with_bnb_linear
   3478     llm_int8_skip_modules = quantization_config.llm_int8_skip_modules
   3479     load_in_8bit_fp32_cpu_offload = quantization_config.llm_int8_enable_fp32_cpu_offload

File <frozen importlib._bootstrap>:1075, in _handle_fromlist(module, fromlist, import_, recursive)

File /usr/local/lib/python3.10/dist-packages/transformers/utils/import_utils.py:1372, in _LazyModule.__getattr__(self, name)
   1370     value = self._get_module(name)
   1371 elif name in self._class_to_module.keys():
-> 1372     module = self._get_module(self._class_to_module[name])
   1373     value = getattr(module, name)
   1374 else:

File /usr/local/lib/python3.10/dist-packages/transformers/utils/import_utils.py:1384, in _LazyModule._get_module(self, module_name)
   1382     return importlib.import_module("." + module_name, self.__name__)
   1383 except Exception as e:
-> 1384     raise RuntimeError(
   1385         f"Failed to import {self.__name__}.{module_name} because of the following error (look up to see its"
   1386         f" traceback):\n{e}"
   1387     ) from e

RuntimeError: Failed to import transformers.integrations.bitsandbytes because of the following error (look up to see its traceback):

        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

$ which nvcc
/usr/local/cuda/bin/nvcc

I've tried to add /usr/local/cuda/bin to LD_LIBRARY_PATH with export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/bin/, but the error remains.

Reproduction

from transformers import AutoTokenizer, AutoModelForCausalLM model = "meta-llama/Llama-2-7b-chat-hf" tokenizer = AutoTokenizer.from_pretrained(model) model = AutoModelForCausalLM.from_pretrained(model, device_map="auto", load_in_4bit=True)

Expected behavior

I should be able to load the model in 4bit.

[josemerinom]

GOOGLE COLAB:

!apt-get update !apt-get install cuda-toolkit-11-8 import os os.environ["LD_LIBRARY_PATH"] += ":" + "/usr/local/cuda-11/lib64" os.environ["LD_LIBRARY_PATH"] += ":" + "/usr/local/cuda-11.8/lib64"

[IamGianluca]

Hi @josemerinom,

I prefer to use CUDA 12.3 if possible. I've tried to add the suggested paths to LD_LIBRARY_PATH but still see the same issue. For reference, here is what I currently have in LD_LIBRARY_PATH if that can ease debugging:

'/usr/local/lib/python3.10/dist-packages/torch/lib:/usr/local/lib/python3.10/dist-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda:/usr/local/cuda/lib64:/usr/local/cuda/bin:/usr/local/cuda-12:/usr/local/cuda-12/lib64:/usr/local/cuda-12.3/:/usr/local/cuda-12.3/lib64'

[IamGianluca]

I just tried installing bitsandbytes via pip on a system with CUDA 12.2, and it works. On a system with CUDA 12.3, I can't get it to work both building the package from source or installing via pip.

[laleye]

I have the same issue on my system with CUDA 12.2. @IamGianluca Which version have installed ?

[IamGianluca]

I can successfully install and use bitsandbytes in a system with CUDA 12.2.

FROM nvcr.io/nvidia/pytorch:23.10-py3
RUN pip install transformers bitsandbytes

In any more recent image (e.g., pytorch:23.11-py3), that ships with CUDA 12.3 installed, I can't install bitsandbytes via pip. In that case, I'm able to pull the repository and successfully build it from source. However, when doing so, I incur in the issue mentioned before.

[SahShk]

I am also facing Same Issue after compilation from source also, i am using python3.11.3, cuda 12.3 , torch 2.1.2, lion-pytorch 0.1.2with centos 7. Please help to resolve issue.

libbitsandbytes_cuda121_nocublaslt.so. is not getting created

===================================BUG REPORT===================================

The following directories listed in your path were found to be non-existent: {PosixPath('vs/workbench/api/node/extensionHostProcess')} The following directories listed in your path were found to be non-existent: {PosixPath('//matplotlib_inline.backend_inline'), PosixPath('module')} CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... DEBUG: Possible options found for libcudart.so: {PosixPath('/usr/local/cuda/lib64/libcudart.so')} CUDA SETUP: PyTorch settings found: CUDA_VERSION=121, Highest Compute Capability: 7.0. CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md CUDA SETUP: Required library version not found: libbitsandbytes_cuda121_nocublaslt.so. Maybe you need to compile it from source? CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...

================================================ERROR===================================== CUDA SETUP: CUDA detection failed! Possible reasons:

1. You need to manually override the PyTorch CUDA version. Please see: "https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
2. CUDA driver not installed
3. CUDA not installed
4. You have multiple conflicting CUDA libraries
5. Required library not pre-compiled for this bitsandbytes release! CUDA SETUP: If you compiled from source, try again with make CUDA_VERSION=DETECTED_CUDA_VERSION for example, make CUDA_VERSION=113. CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version via conda list | grep cuda. ================================================================================

CUDA SETUP: Something unexpected happened. Please compile from source: git clone https://github.com/TimDettmers/bitsandbytes.git cd bitsandbytes CUDA_VERSION=121_nomatmul python setup.py install CUDA SETUP: Setup Failed!

[SoniaGrh]

I am also facing Same Issue after compilation from source also, i am using python3.11.3, cuda 12.3 , torch 2.1.2, lion-pytorch 0.1.2with centos 7. Please help to resolve issue.

libbitsandbytes_cuda121_nocublaslt.so. is not getting created

===================================BUG REPORT===================================

The following directories listed in your path were found to be non-existent: {PosixPath('vs/workbench/api/node/extensionHostProcess')} The following directories listed in your path were found to be non-existent: {PosixPath('//matplotlib_inline.backend_inline'), PosixPath('module')} CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... DEBUG: Possible options found for libcudart.so: {PosixPath('/usr/local/cuda/lib64/libcudart.so')} CUDA SETUP: PyTorch settings found: CUDA_VERSION=121, Highest Compute Capability: 7.0. CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md CUDA SETUP: Required library version not found: libbitsandbytes_cuda121_nocublaslt.so. Maybe you need to compile it from source? CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...

================================================ERROR===================================== CUDA SETUP: CUDA detection failed! Possible reasons:

1. You need to manually override the PyTorch CUDA version. Please see: "https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
2. CUDA driver not installed
3. CUDA not installed
4. You have multiple conflicting CUDA libraries

5. Required library not pre-compiled for this bitsandbytes release! CUDA SETUP: If you compiled from source, try again with make CUDA_VERSION=DETECTED_CUDA_VERSION for example, make CUDA_VERSION=113. CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version via conda list | grep cuda.

CUDA SETUP: Something unexpected happened. Please compile from source: git clone https://github.com/TimDettmers/bitsandbytes.git cd bitsandbytes CUDA_VERSION=121_nomatmul python setup.py install CUDA SETUP: Setup Failed!

I had the same issue, I upgraded the version of bitsandbytes to 0.42.0 and it worked.

[SahShk]

Yes i am using newly updated bitsandbytes 0.42.0 compilled it from source but still facing same issue. Not able to traceback error or what is the problem. Also I tried make CUDA_VERSION=121 it gives error as ".....error: macro "NV_IF_TARGET" passed 3 arguments, but takes just 2 binary_op);)); ^ make: *** [Makefile:60: all] Error 1"

for check:

python -m bitsandbytes

False

===================================BUG REPORT=================================== /VM_Data/Generative_AI_test/bitsandbytes/bitsandbytes/cuda_setup/main.py:167: UserWarning: Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

warn(msg)

/VM_Data/Generative_AI_test/bitsandbytes/bitsandbytes/cuda_setup/main.py:167: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda-12.3/libcudart.so'), PosixPath('/usr/local/openssl/lib/libcudart.so')}.. We select the PyTorch default libcudart.so, which is {torch.version.cuda},but this might missmatch with the CUDA version that is needed for bitsandbytes.To override this behavior set the BNB_CUDA_VERSION=<version string, e.g. 122> environmental variableFor example, if you want to use the CUDA version 122BNB_CUDA_VERSION=122 python ...OR set the environmental variable in your .bashrc: export BNB_CUDA_VERSION=122In the case of a manual override, make sure you set the LD_LIBRARY_PATH, e.g.export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.2 warn(msg) /VM_Data/Generative_AI_test/bitsandbytes/bitsandbytes/cuda_setup/main.py:167: UserWarning: /usr/local/openssl/lib:/usr/local/openssl/lib:/usr/local/openssl/lib::/usr/local/cuda-12.3 did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths... warn(msg) CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... DEBUG: Possible options found for libcudart.so: {PosixPath('/usr/local/cuda/lib64/libcudart.so')} CUDA SETUP: PyTorch settings found: CUDA_VERSION=121, Highest Compute Capability: 7.0. CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md /VM_Data/Generative_AI_test/bitsandbytes/bitsandbytes/cuda_setup/main.py:167: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU! If you run into issues with 8-bit matmul, you can try 4-bit quantization: https://huggingface.co/blog/4bit-transformers-bitsandbytes warn(msg) CUDA SETUP: Required library version not found: libbitsandbytes_cuda121_nocublaslt.so. Maybe you need to compile it from source? CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...

================================================ERROR===================================== CUDA SETUP: CUDA detection failed! Possible reasons:

1. You need to manually override the PyTorch CUDA version. Please see: "https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md
2. CUDA driver not installed
3. CUDA not installed
4. You have multiple conflicting CUDA libraries
5. Required library not pre-compiled for this bitsandbytes release! CUDA SETUP: If you compiled from source, try again with make CUDA_VERSION=DETECTED_CUDA_VERSION for example, make CUDA_VERSION=113. CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version via conda list | grep cuda. ================================================================================

CUDA SETUP: Something unexpected happened. Please compile from source: git clone https://github.com/TimDettmers/bitsandbytes.git cd bitsandbytes CUDA_VERSION=121_nomatmul python setup.py install CUDA SETUP: Setup Failed! Traceback (most recent call last): File "", line 189, in _run_module_as_main File "", line 148, in _get_module_details File "", line 112, in _get_module_details File "/VM_Data/Generative_AI_test/bitsandbytes/bitsandbytes/init.py", line 6, in from . import cuda_setup, utils, research File "/VM_Data/Generative_AI_test/bitsandbytes/bitsandbytes/research/init.py", line 1, in from . import nn File "/VM_Data/Generative_AI_test/bitsandbytes/bitsandbytes/research/nn/init.py", line 1, in from .modules import LinearFP8Mixed, LinearFP8Global File "/VM_Data/Generative_AI_test/bitsandbytes/bitsandbytes/research/nn/modules.py", line 8, in from bitsandbytes.optim import GlobalOptimManager File "/VM_Data/Generative_AI_test/bitsandbytes/bitsandbytes/optim/init.py", line 6, in from bitsandbytes.cextension import COMPILED_WITH_CUDA File "/VM_Data/Generative_AI_test/bitsandbytes/bitsandbytes/cextension.py", line 20, in raise RuntimeError(''' RuntimeError: CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

[DanielKim15]

I too have the same problem! I've pip installed it onto my virtual environment and i triple checked my PATH for my CUDA library, but it seems the problem still continues. Currently I am using python3.11.3, cuda 11.8 , torch 2.1.2+cu118. Is there a way I can fix this?

===================================BUG REPORT===================================

The following directories listed in your path were found to be non-existent: {WindowsPath('AQAAANCMnd8BFdERjHoAwE/Cl+sBAAAAGM6/x5/Irk6FENB/hhKDMAQAAAACAAAAAAAQZgAAAAEAACAAAAA6DcBej+z/GNccXl+p+rfboWsa+8VjZq7vengAtnrPnAAAAAAOgAAAAAIAACAAAABg8++o9nrqYEnjyDTIP/tsUrXJLFKWwX0IDqsiakjVymAAAADI+L8dQC1K/r0d8iZ05+L5P8jhdZXSf4re8ecdxhZU1d9Yg1NlFWbc0e9tpzrFmRNSYrUywdySaDwB2KTl4B/DNrplPjb1tzbp2B1XuJRrxf/ygOcmT+TdEDSdkGh0Q1dAAAAAS5aEJuC9y4vHvHRgQA4/7fPEM2BGiDYGWqS0+pa3IhyDb4xeWEvmsGINoaH6neX0PVKBJRoYsh8FvpAYAGc7Ag==')} The following directories listed in your path were found to be non-existent: {WindowsPath('vs/workbench/api/node/extensionHostProcess')} The following directories listed in your path were found to be non-existent: {WindowsPath('module'), WindowsPath('/matplotlib_inline.backend_inline')} CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')} DEBUG: Possible options found for libcudart.so: set() CUDA SETUP: PyTorch settings found: CUDA_VERSION=118, Highest Compute Capability: 8.6. CUDA SETUP: To manually override the PyTorch CUDA version please see:https://github.com/TimDettmers/bitsandbytes/blob/main/how_to_use_nonpytorch_cuda.md CUDA SETUP: Loading binary c:\Users\dkim.CENSEO\OneDrive - Censeo Consulting Group\Desktop\python_environment_llm.venv\Lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.so... argument of type 'WindowsPath' is not iterable CUDA SETUP: Problem: The main issue seems to be that the main CUDA runtime library was not detected. CUDA SETUP: Solution 1: To solve the issue the libcudart.so location needs to be added to the LD_LIBRARY_PATH variable CUDA SETUP: Solution 1a): Find the cuda runtime library via: find / -name libcudart.so 2>/dev/null CUDA SETUP: Solution 1b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_1a CUDA SETUP: Solution 1c): For a permanent solution add the export from 1b into your .bashrc file, located at ~/.bashrc CUDA SETUP: Solution 2: If no library was found in step 1a) you need to install CUDA. CUDA SETUP: Solution 2a): Download CUDA install script: wget https://raw.githubusercontent.com/TimDettmers/bitsandbytes/main/cuda_install.sh CUDA SETUP: Solution 2b): Install desired CUDA version to desired location. The syntax is bash cuda_install.sh CUDA_VERSION PATH_TO_INSTALL_INTO. CUDA SETUP: Solution 2b): For example, "bash cuda_install.sh 113 ~/local/" will download CUDA 11.3 and install into the folder ~/local c:\Users\dkim.CENSEO\OneDrive - Censeo Consulting Group\Desktop\python_environment_llm.venv\Lib\site-packages\bitsandbytes\cuda_setup\main.py:167: UserWarning: Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

warn(msg)

RuntimeError Traceback (most recent call last) Cell In[5], line 6 1 #loads up the model 2 #base_model_id = "C:/Users/dkim.CENSEO/OneDrive - Censeo Consulting Group/Desktop/mistral_7b_instruct_v2" 3 #model = AutoModelForCausalLM.from_pretrained(base_model_id) 5 from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig ----> 6 from bitsandbytes import BitsAndBytesConfig 8 #fix this! 9 base_model_id = "C:/Users/dkim.CENSEO/OneDrive - Censeo Consulting Group/Desktop/mistral_7b_instruct_v2"

File c:\Users\dkim.CENSEO\OneDrive - Censeo Consulting Group\Desktop\python_environment_llm.venv\Lib\site-packages\bitsandbytes_init_.py:6 1 # Copyright (c) Facebook, Inc. and its affiliates. 2 # 3 # This source code is licensed under the MIT license found in the 4 # LICENSE file in the root directory of this source tree. ----> 6 from . import cuda_setup, utils, research 7 from .autograd._functions import ( 8 MatmulLtState, 9 bmm_cublas, (...) 13 matmul_4bit 14 ) 15 from .cextension import COMPILED_WITH_CUDA

File c:\Users\dkim.CENSEO\OneDrive - Censeo Consulting Group\Desktop\python_environment_llm.venv\Lib\site-packages\bitsandbytes\research_init_.py:1 ----> 1 from . import nn 2 from .autograd._functions import ( 3 switchback_bnb, 4 matmul_fp8_global, 5 matmul_fp8_mixed, 6 )

File c:\Users\dkim.CENSEO\OneDrive - Censeo Consulting Group\Desktop\python_environment_llm.venv\Lib\site-packages\bitsandbytes\research\nn_init_.py:1 ----> 1 from .modules import LinearFP8Mixed, LinearFP8Global

File c:\Users\dkim.CENSEO\OneDrive - Censeo Consulting Group\Desktop\python_environment_llm.venv\Lib\site-packages\bitsandbytes\research\nn\modules.py:8 5 from torch import Tensor, device, dtype, nn 7 import bitsandbytes as bnb ----> 8 from bitsandbytes.optim import GlobalOptimManager 9 from bitsandbytes.utils import OutlierTracer, find_outlier_dims 11 T = TypeVar("T", bound="torch.nn.Module")

File c:\Users\dkim.CENSEO\OneDrive - Censeo Consulting Group\Desktop\python_environment_llm.venv\Lib\site-packages\bitsandbytes\optim_init_.py:6 1 # Copyright (c) Facebook, Inc. and its affiliates. 2 # 3 # This source code is licensed under the MIT license found in the 4 # LICENSE file in the root directory of this source tree. ----> 6 from bitsandbytes.cextension import COMPILED_WITH_CUDA 8 from .adagrad import Adagrad, Adagrad8bit, Adagrad32bit 9 from .adam import Adam, Adam8bit, Adam32bit, PagedAdam, PagedAdam8bit, PagedAdam32bit

File c:\Users\dkim.CENSEO\OneDrive - Censeo Consulting Group\Desktop\python_environment_llm.venv\Lib\site-packages\bitsandbytes\cextension.py:20 18 CUDASetup.get_instance().generate_instructions() 19 CUDASetup.get_instance().print_log_stack() ---> 20 raise RuntimeError(''' 21 CUDA Setup failed despite GPU being available. Please run the following command to get more information: 22 23 python -m bitsandbytes 24 25 Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them 26 to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes 27 and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues''') 28 lib.cadam32bit_grad_fp32 # runs on an error if the library could not be found -> COMPILED_WITH_CUDA=False 29 lib.get_context.restype = ct.c_void_p

RuntimeError: CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

[Quetzalcohuatl]

Judging by the newness of this issue, maybe it's because bitsandbytes does not support cuda 12.3? What if you downgrade cuda

[SahShk]

i resolved my problem, its now running. Things to do are: first check gcc version. If it is upgraded run "strings /usr/lib64/libstdc++.so.6 | grep 'CXXABI'" or else update gcc. After updatation check CXXABI to be compatible. Other things you need to verify is versions of libraries like " !pip install -q torch peft==0.4.0 bitsandbytes==0.42.0 transformers==4.31.0 trl==0.4.7 accelerate". It should work now

[Parvez-Khan-1]

I just tried installing bitsandbytes via pip on a system with CUDA 12.2, and it works. On a system with CUDA 12.3, I can't get it to work both building the package from source or installing via pip.

Pretty much the same behaviour for me, I cant get it work both building the package from source or installing via pip. Any suggestions or inputs?