Can it be run on Windows? - Issue with Transformers and bitsandbytes

[chgayot]

I'm trying to set it up on Windows, and while I had no problem to set up CUDA ( python returns True to torch.cuda.is_available() ) But it seems that the transformers library (taken from git at the moment) uses directly bitsandbytes when loading the model, and as is it Linux only, it returns an error, see below. I couldn't find any related issue on the transformers issue tracker, maybe you have a solution or a commit where it was working on Windows?

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin C:\Programs\Anaconda\envs\ai-falcon\Lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so
False
C:\Programs\Anaconda\envs\ai-falcon\Lib\site-packages\bitsandbytes\cuda_setup\main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/Programs/Anaconda/envs/ai-falcon/lib'), WindowsPath('C')}
  warn(msg)
C:\Programs\Anaconda\envs\ai-falcon\Lib\site-packages\bitsandbytes\cuda_setup\main.py:149: UserWarning: C:\Programs\Anaconda\envs\ai-falcon did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
  warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
C:\Programs\Anaconda\envs\ai-falcon\Lib\site-packages\bitsandbytes\cuda_setup\main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
  warn(msg)
CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed? If you are on a cluster, make sure you are on a CUDA machine!
C:\Programs\Anaconda\envs\ai-falcon\Lib\site-packages\bitsandbytes\cuda_setup\main.py:149: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
  warn(msg)
C:\Programs\Anaconda\envs\ai-falcon\Lib\site-packages\bitsandbytes\cuda_setup\main.py:149: UserWarning: WARNING: No GPU detected! Check your CUDA paths. Proceeding to load CPU-only library...
  warn(msg)
CUDA SETUP: Loading binary C:\Programs\Anaconda\envs\ai-falcon\Lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA library was not detected.
CUDA SETUP: Solution 1): Your paths are probably not up-to-date. You can update them via: sudo ldconfig.
CUDA SETUP: Solution 2): If you do not have sudo rights, you can do the following:
CUDA SETUP: Solution 2a): Find the cuda library via: find / -name libcuda.so 2>/dev/null
CUDA SETUP: Solution 2b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_2a
CUDA SETUP: Solution 2c): For a permanent solution add the export from 2b into your .bashrc file, located at ~/.bashrc
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "D:\StepUpSolutionsGit\falcon-play\scripts\chat_play.py", line 318, in <module>
    main()
  File "D:\StepUpSolutionsGit\falcon-play\scripts\chat_play.py", line 196, in main
    model: AutoModelForCausalLM = get_model(model_args)
                                  ^^^^^^^^^^^^^^^^^^^^^
  File "D:\StepUpSolutionsGit\falcon-play\scripts\chat_play.py", line 176, in get_model
    model = AutoModelForCausalLM.from_pretrained(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Programs\Anaconda\envs\ai-falcon\Lib\site-packages\transformers\models\auto\auto_factory.py", line 467, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Programs\Anaconda\envs\ai-falcon\Lib\site-packages\transformers\modeling_utils.py", line 2642, in from_pretrained
    from .utils.bitsandbytes import get_keys_to_not_convert, replace_with_bnb_linear
  File "C:\Programs\Anaconda\envs\ai-falcon\Lib\site-packages\transformers\utils\bitsandbytes.py", line 10, in <module>
    import bitsandbytes as bnb
  File "C:\Programs\Anaconda\envs\ai-falcon\Lib\site-packages\bitsandbytes\__init__.py", line 6, in <module>
    from . import cuda_setup, utils, research
  File "C:\Programs\Anaconda\envs\ai-falcon\Lib\site-packages\bitsandbytes\research\__init__.py", line 1, in <module>
    from . import nn
  File "C:\Programs\Anaconda\envs\ai-falcon\Lib\site-packages\bitsandbytes\research\nn\__init__.py", line 1, in <module>
    from .modules import LinearFP8Mixed, LinearFP8Global
  File "C:\Programs\Anaconda\envs\ai-falcon\Lib\site-packages\bitsandbytes\research\nn\modules.py", line 8, in <module>
    from bitsandbytes.optim import GlobalOptimManager
  File "C:\Programs\Anaconda\envs\ai-falcon\Lib\site-packages\bitsandbytes\optim\__init__.py", line 6, in <module>
    from bitsandbytes.cextension import COMPILED_WITH_CUDA
  File "C:\Programs\Anaconda\envs\ai-falcon\Lib\site-packages\bitsandbytes\cextension.py", line 20, in <module>
    raise RuntimeError('''
RuntimeError:
        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues