Missing Windows support

[TimDettmers]

Currently, the library uses precompiled Linux binaries. I am unsure how compatible these are with standard PyTorch installs on Windows. It might be that the binaries need to be compiled against mingw32/64 to create functional binaries for Windows.

The most helpful would be a case where a person is able to compile from source and use the library. This will require altering the Makefile file. If this works, we can add instructions on compiling for Windows as a first step before doing a full-scale Windows deployment of binaries on pip.

Since I do not have a Windows machine, any help is wanted on this!

[DeXtmL]

I am able to compile the csrc part on Windows but fail to link (Visual Studio) FYI, this is the error message. I'm no expert on cuda, so I'm not quite sure where it goes wrong.

To successfully compile on windows MSVC, some parts need to be patched as following:

1. rename pythonInterface.c to pythonInterface.cpp, or visual studio will try using a C compiler for it.

2. add one missing template instantiation like this: (in SIMD.h)

3. get unistd.h and getopt.h for windows

4. get pthread for windows

finally, this is just a build test, so I'm not using the Makefile. That's all for now. Tested on CUDA toolkit 11.6, windows 11

[DeXtmL]

After some really tedious debugging and tackling various hidden problems, I managed to compile the whole module. This is the end result: https://github.com/DeXtmL/bitsandbytes-win-prebuilt

The binaries are compiled against CUDAToolkit 11.6 and Visual Studio 2022. I am able to make inferences nearly identical to "normal" fp16 version. So this is kind of a confirmation for "it works" shout-out. No vigorous testing was conducted though. @TimDettmers Finally, the "cuda_setup" part of the source code is entirely incompatible with Windows, there are loads of hardcoded routines; so I used a quick makeshift patch instead of making it proper, that's also why I'm not posting my changes or making PR for now. If you are eager to test:

in cuda_setup/main.py:
make evaluate_cuda_setup() always return "libbitsandbytes_cuda116.dll"
in ./cextension.py:
change ct.cdll.LoadLibrary(binary_path) to ct.cdll.LoadLibrary(str(binary_path))

That should do the trick.

Hopefully this can help someone in the Windows territory; let's hope the official windows support come fast.

[TemporalLabsLLC-SOL]

Thank you. This might be the key.

[korakoe]

After some really tedious debugging and tackling various hidden problems, I managed to compile the whole module. This is the end result: https://github.com/DeXtmL/bitsandbytes-win-prebuilt

The binaries are compiled against CUDAToolkit 11.6 and Visual Studio 2022. I am able to make inferences nearly identical to "normal" fp16 version. So this is kind of a confirmation for "it works" shout-out. No vigorous testing was conducted though. @TimDettmers Finally, the "cuda_setup" part of the source code is entirely incompatible with Windows, there are loads of hardcoded routines; so I used a quick makeshift patch instead of making it proper, that's also why I'm not posting my changes or making PR for now. If you are eager to test:

in cuda_setup/main.py:
make evaluate_cuda_setup() always return "libbitsandbytes_cuda116.dll"
in ./cextension.py:
change ct.cdll.LoadLibrary(binary_path) to ct.cdll.LoadLibrary(str(binary_path))

That should do the trick.

Hopefully this can help someone in the Windows territory; let's hope the official windows support come fast.

could you provide a makefile for this?

[bmaltais]

After some really tedious debugging and tackling various hidden problems, I managed to compile the whole module. This is the end result: https://github.com/DeXtmL/bitsandbytes-win-prebuilt

The binaries are compiled against CUDAToolkit 11.6 and Visual Studio 2022. I am able to make inferences nearly identical to "normal" fp16 version. So this is kind of a confirmation for "it works" shout-out. No vigorous testing was conducted though. @TimDettmers Finally, the "cuda_setup" part of the source code is entirely incompatible with Windows, there are loads of hardcoded routines; so I used a quick makeshift patch instead of making it proper, that's also why I'm not posting my changes or making PR for now. If you are eager to test:

in cuda_setup/main.py:
make evaluate_cuda_setup() always return "libbitsandbytes_cuda116.dll"
in ./cextension.py:
change ct.cdll.LoadLibrary(binary_path) to ct.cdll.LoadLibrary(str(binary_path))

That should do the trick.

Hopefully this can help someone in the Windows territory; let's hope the official windows support come fast.

Where do you put the pre-built file to activate adam?

[davidvfx07]

After some really tedious debugging and tackling various hidden problems, I managed to compile the whole module. This is the end result: https://github.com/DeXtmL/bitsandbytes-win-prebuilt

The binaries are compiled against CUDAToolkit 11.6 and Visual Studio 2022. I am able to make inferences nearly identical to "normal" fp16 version. So this is kind of a confirmation for "it works" shout-out. No vigorous testing was conducted though. @TimDettmers Finally, the "cuda_setup" part of the source code is entirely incompatible with Windows, there are loads of hardcoded routines; so I used a quick makeshift patch instead of making it proper, that's also why I'm not posting my changes or making PR for now. If you are eager to test:

in cuda_setup/main.py:
make evaluate_cuda_setup() always return "libbitsandbytes_cuda116.dll"
in ./cextension.py:
change ct.cdll.LoadLibrary(binary_path) to ct.cdll.LoadLibrary(str(binary_path))

That should do the trick.

Hopefully this can help someone in the Windows territory; let's hope the official windows support come fast.

An easy way to always return libbitsandbytes_cuda116.dll would be to insert if torch.cuda.is_available(): return 'libbitsandbytes_cuda116.dll', None, None, None, None above 119: if not torch.cuda.is_available(): return 'libsbitsandbytes_cpu.so', None, None, None, None

[davidvfx07]

After some really tedious debugging and tackling various hidden problems, I managed to compile the whole module. This is the end result: https://github.com/DeXtmL/bitsandbytes-win-prebuilt The binaries are compiled against CUDAToolkit 11.6 and Visual Studio 2022. I am able to make inferences nearly identical to "normal" fp16 version. So this is kind of a confirmation for "it works" shout-out. No vigorous testing was conducted though. @TimDettmers Finally, the "cuda_setup" part of the source code is entirely incompatible with Windows, there are loads of hardcoded routines; so I used a quick makeshift patch instead of making it proper, that's also why I'm not posting my changes or making PR for now. If you are eager to test:

in cuda_setup/main.py:
make evaluate_cuda_setup() always return "libbitsandbytes_cuda116.dll"
in ./cextension.py:
change ct.cdll.LoadLibrary(binary_path) to ct.cdll.LoadLibrary(str(binary_path))

That should do the trick. Hopefully this can help someone in the Windows territory; let's hope the official windows support come fast.

Where do you put the pre-built file to activate adam?

You put them in site-packages\bitsandbytes

[TinyBeeman]

Appreciate you doing this work, helped unblock me in a big way. I hope bitsandbytes supports Windows directly sooner rather than later, but this is a great stopgap.

[desis123]

@PinPointPing Thanks a lot, it just worked like a charm

[FloydianSound]

Same issue for me, if someone can compile cuda 11.8 binaries for me i can test them :) using StableDiffusion-WebUI + Dreambooth, would love to give Adam a spin !

btw under Windows, the environment variable is CUDA_PATH and CUDA_PATH_V11_8

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
For effortless bug reporting copy-paste your error into this form: https://docs.google.com/forms/d/e/1FAIpQLScPB8emS3Thkp66nvqwmjTEgxp8Y9ufuWTzFyr9kJ5AoI47dQ/viewform?usp=sf_link
================================================================================
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
CUDA SETUP: Loading binary G:\Visions of Chaos\MachineLearning\Text To Image\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so...
Exception importing 8bit adam: argument of type 'WindowsPath' is not iterable
 Scheduler Loaded 
 Allocated: 2.3GB 
 Reserved: 2.4GB 

[FloydianSound]

after following some advice and making the edits in this thread, i got Adam to run on my CUDA 11.8 setup/ it does indeed work for me :)

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
For effortless bug reporting copy-paste your error into this form: https://docs.google.com/forms/d/e/1FAIpQLScPB8emS3Thkp66nvqwmjTEgxp8Y9ufuWTzFyr9kJ5AoI47dQ/viewform?usp=sf_link
================================================================================
CUDA SETUP: Loading binary G:\Visions of Chaos\MachineLearning\Text To Image\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda116.dll...
 Scheduler Loaded 
 Allocated: 0.3GB 
 Reserved: 0.4GB 

[geocine]

would you mind sharing your cuda 11.8 binary

[FloydianSound]

would you mind sharing your cuda 11.8 binary

using the libs cuda 11.6 from [(https://github.com/DeXtmL/bitsandbytes-win-prebuilt)] and cuda 11.8 on windows 10

[FloydianSound]

@TimDettmers any updates on this ? 😄

[centerionware]

I just got this building with cmake.

First thing I did was make a directory called dependencies , then download https://github.com/GerHobbelt/pthread-win32 and extract the whole thing there so it's /project_root/dependencies/pthread-win32-main/pthread.h

(I lied, I didn't really do that first, did that last but I suggest anyone following this do it first to avoid the error)

I did apply the patch above by @DeXtmL (Does not compile without it, missing vec_t error). As for unistd.h/getopt, I literally just commented out the #include <unistd.h> in the file it gave the error on and it now fully compiles so not sure anything relies on unistd.h (At least in windows with VC2019.. ), we'll see when I go to test the .dll (Didn't give any errors about getopt)

~CMakeLists.txt~ ~CMakeLists.txt~ **** (New files in next reply) ****

The first CMakeLists.txt is in the root, the second in csrc. I do:

cd /project_root
mkdir build
cd build
cmake ..

Then can open the .sln, right click bitsandbytes hit build and it goes to town.

I imagine for linux the same cmake file would actually work just fine. Only thing I did special for windows was add the include path for the pthreads, adding an include path to a folder that doesn't exist probably won't hurt right?

GL

[centerionware]

Quick final follow up, it built fine in Debug with the above, i actually don't know quite how. For release mode I did actually have to build the pthread library (just another mkdir build, cmake .., open solution, build all in release mode), then slightly modify the cmake file. (Probably could have just done cmake .. ; cmake --build . -j4 --config Release ; to build pthread) ~CMakeLists.txt~

I don't know why it worked before in debug mode at all, because I had link_libraries wrong (don't have the -l in cmake in front), and for release mode had to fix that and include pthreadVC3.lib .. here's the final final from me CMakeLists.txt for the csrc folder ~CMakeLists.txt~ **EDIT I got it to load up, but it says it compiled without GPU support so i'm still working on it. **EDIT2 Still working on it. Added "add_compile_definitions(BUILD_CUDA)" then checked the resulting vc files and it does enable the BUILD_CUDA define, and in pythonInterface.cpp visual studio says BUILD_CUDA is defined, I can see where the cadam32bit_g32 is generated via that macro, but not quite sure why when it loads the .dll - lib.cadam32bit_g32 throws an attribute error. **EDIT3 I finally got it to work. It took a couple hours (long compile times) but I finally got one that exports all symbols. the trick was putting the thing in a different cmake file ffs. The final two root/CMakeLists.txt CMakeLists.txt

root/csrc/CMakeLists.txt CMakeLists.txt

mkdir build, cd build, cmake .. , cmake --build ./ -j4 --config Release , .dll is put into build/csrc/Release/bitsandbyts.dll

just for fun here's it running gpt-j-6b on a RTX3080 on windows 11 with cuda 113

import torch
import transformers

access_token = "hf_"

from transformers.models.gptj import GPTJForCausalLM


device = 'cuda' if torch.cuda.is_available() else 'cpu'

tokenizer = transformers.AutoTokenizer.from_pretrained("EleutherAI/gpt-j-6B", use_auth_token=access_token, device_map='auto')
gpt = GPTJForCausalLM.from_pretrained("EleutherAI/gpt-j-6B", use_auth_token=access_token, device_map='auto', load_in_8bit=True, low_cpu_mem_usage=True).to(device)

prompt = tokenizer("A cat sat on a mat", return_tensors='pt')
prompt = {key: value.to(device) for key, value in prompt.items()}
out = gpt.generate(**prompt, min_length=128, max_length=128, do_sample=True)
tokenizer.decode(out[0])

>>> tokenizer.decode(out[0])
"A cat sat on a mat, staring at me, his back legs tucked under him, tail swerving in quick little circles.\n\nI squatted next to him and leaned against the cold wooden wall. I'd come down here to feed the cat, but I'd been too tired and cold and my stomach still ached and my hands and feet were numb from spending the night in a tree. Besides, this was not my house, not my town, not my time. The cat stared at me, his green eyes the only proof he knew I was the intruder he was protecting.\n\nI wished I'd brought a blanket"

[james-things]

Quick final follow up...

You are a hero, thank you so much for posting this!

[ZQ-Dev8]

Quick final follow up...

Any chance you could hang some instructions somewhere to help others replicate the process? I'm trying to follow along but have run into some issues. A step-by-step would be awesome.

[centerionware]

From my not so great memory it's something like:

To build the bitsandbytes project for Windows, you will need two programs: cmake and nvcc. You can a build environment such as Visual Studio and Miniconda.

Open the command line interface (CLI) for your build environment. (Start menu/visual studio/ one of them consoles) Activate your chosen environment(Miniconda) and install necessary packages. ( cuda-nvcc iirc? Probably a cuda environment like https://pytorch.org/get-started/locally/ ) Place cmake files in the right location. Build pthreads (if necessary) using cmake (The same commands as below). Edit1 ( download https://github.com/GerHobbelt/pthread-win32 and extract the whole thing there so it's /project_root/dependencies/pthread-win32-main/pthread.h ) END EDIT1

Run the following commands: (Note -j4 means use 4 cores to build. If you don't have 4 cores, or you have a lot more, change that number.)

(Assuming on C:\ drive, if on other drive, change letter on first and second line appropriately.

C:
cd C:\PROJECT_ROOT\  ( or cd C:\PROJECT_ROOT\dependencies\pthread-win32-main )
mkdir build
cd build
cmake ..
cmake --build ./ -j4 --config Release

The resulting dll file will be in build/csrc/Release/bitsandbytes.dll. Edit2 When it errors about unistd.h or getopt.h, open that file and comment out the #include - although a more proper way would to be change to detect _MSC_VER, if found just don't include unistd.h ( wouldn't test against WIN32 cause can be true in mingw, WSL,(etc) environment where using unistd.h would be required still, where _MSC_VER indicates the microsoft visual studio compiler version ) EG:

#ifndef _MSC_VER
#include <unistd.h>
#endif

END EDIT2

[healthyfat]

Is it possible to run bitsandbytes with 2060RTX 6GB on Windows 10?

[centerionware]

Is it possible to run bitsandbytes with 2060RTX 6GB on Windows 10?

I don't see why it wouldn't run on a 2060, just be aware it doesn't eliminate vram requirements, just reduces them. Still wouldn't be able to run chatgpt for example with it's 800GB+ 32-bit precision vram requirement (if had access to that). Any model that takes <24gb of vram in 32bit, or <12gb of vram in 16bit mode, should be able to fit in 6gb at 8bit.

[healthyfat]

@centerionware

Thanks for the answer.

I think I get confused about bitsandbytes. I was thinking it only works on Linux and repository such as kohya-ss/sd-scripts or bmaltais/kohya_ss won't work on Windows, because of the lack of compatibility and that is the reason I am not able to run them. But this is not true, as I manage to make them work on my machine eventually.

[DFin]

After some really tedious debugging and tackling various hidden problems, I managed to compile the whole module. This is the end result: https://github.com/DeXtmL/bitsandbytes-win-prebuilt

The binaries are compiled against CUDAToolkit 11.6 and Visual Studio 2022. I am able to make inferences nearly identical to "normal" fp16 version. So this is kind of a confirmation for "it works" shout-out. No vigorous testing was conducted though. @TimDettmers Finally, the "cuda_setup" part of the source code is entirely incompatible with Windows, there are loads of hardcoded routines; so I used a quick makeshift patch instead of making it proper, that's also why I'm not posting my changes or making PR for now. If you are eager to test:

in cuda_setup/main.py:
make evaluate_cuda_setup() always return "libbitsandbytes_cuda116.dll"
in ./cextension.py:
change ct.cdll.LoadLibrary(binary_path) to ct.cdll.LoadLibrary(str(binary_path))

That should do the trick.

Hopefully this can help someone in the Windows territory; let's hope the official windows support come fast.

This solution is still valid and the linked binaries work with cuda 11.7 as well (at least for Adam 8 bit and on win 10). But the location of ct.cdll.LoadLibrary changed and it can now be found in ./cuda_setup/main.py as well and not in ./cextension.py

just replace both occurrences of self.lib = ct.cdll.LoadLibrary(binary_path) with self.lib = ct.cdll.LoadLibrary(str(binary_path))

[underlines]

To use this with facebook-research/LLaMA-7b within text-generation-webui on windows 11:

1. git pull oobabooga/text-generation-webui
2. follow the installation instructions for conda
3. download HuggingFace converted model weights for LLaMA, or convert them by yourself from the original weights. Both leaked on torrent and even on the official facebook llama repo as an unapproved PR.
4. copy the llama-7b folder (or whatever size you want to run) into text-generation-webui\models. The folder should contain the config.json, generation_config.json, pytorch_model.bin.index.json, special_tokens_map.json, tokenizer.model, tokenizer_config.json as well as all the 33 pytorch_model-000xx-of-00033.bin files
5. put libbitsandbytes_cuda116.dll in C:\Users\xxx\miniconda3\envs\textgen\lib\site-packages\bitsandbytes\
6. edit \bitsandbytes\cuda_setup\main.py:

search for: if not torch.cuda.is_available(): return 'libsbitsandbytes_cpu.so', None, None, None, None replace with: if torch.cuda.is_available(): return 'libbitsandbytes_cuda116.dll', None, None, None, None

search for this twice: self.lib = ct.cdll.LoadLibrary(binary_path) replace with: self.lib = ct.cdll.LoadLibrary(str(binary_path))

[testFaze]

I've tried all of the above and I'm still getting an error ... bitsandbytes\libbitsandbytes_cuda116.dll... [WinError 193] %1 is not a valid Win32 application

Any ideas? I thought the whole point of this .dll was that it is a windows version?

[TinyBeeman]

That sounds like an error with a script, not an error with the DLL.

On Wed, Mar 15, 2023 at 5:00 PM testFaze @.***> wrote:

I've tried all of the above and I'm still getting an error ... bitsandbytes\libbitsandbytes_cuda116.dll... [WinError 193] %1 is not a valid Win32 application

Any ideas? I thought the whole point of this .dll was that it is a windows version?

— Reply to this email directly, view it on GitHub https://github.com/TimDettmers/bitsandbytes/issues/30#issuecomment-1471001081, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM3LWCXG5CYX53ELQBR4FQDW4JJZBANCNFSM6AAAAAAQFJ4QZQ . You are receiving this because you commented.Message ID: @.***>

-- Tony Beeman he/him or they/them Assistant Artistic Director | Unexpected Productions http://unexpectedproductions.org/

[testFaze]

Thankyou.

TIL (1) - if you right click/save a filename from GitHub, while it appears you are saving the target file, you're not. The file that appears in the target folder has the same name but is gibberish.

TIL (2) - I am not very good at IT.

All working fine now.

[nullnuller]

What changes are needed to run it on a Windows CPU only machine? Trying to run Llama-Alpaca-LoRa but getting issues with bitsandbytes:

===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64... CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed? If you are on a cluster, make sure you are on a CUDA machine! CUDA SETUP: Loading binary C:\Users\AMahmood\Downloads\Llama-Alpaca-LoRa\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so... [WinError 193] %1 is not a valid Win32 application CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64... CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed? If you are on a cluster, make sure you are on a CUDA machine! CUDA SETUP: Loading binary C:\Users\AMahmood\Downloads\Llama-Alpaca-LoRa\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so... [WinError 193] %1 is not a valid Win32 application .\Llama-Alpaca-LoRa\venv\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')} warn(msg) .\Llama-Alpaca-LoRa\venv\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)! warn(msg) .\Llama-Alpaca-LoRa\venv\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: No GPU detected! Check your CUDA paths. Proceeding to load CPU-only library... warn(msg) .\Llama-Alpaca-LoRa\venv\lib\site-packages\bitsandbytes\cextension.py:31: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable. warn("The installed version of bitsandbytes was compiled without GPU support. " .\Llama-Alpaca-LoRa\venv\Lib\site-packages\bitsandbytes\cextension.py:31: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable. warn("The installed version of bitsandbytes was compiled without GPU support. "

Process finished with exit code 0

[acpopescu]

I've got this compiling under CUDA 11.7 with CMake if y'all are interested. I DID NOT RUN ANY TESTS yet, it is too late in the day

Prototype CMAKE file, it is missing functionality of the makefile. It is usable to target a single config, and does not bring in /dependencies/cub.

https://github.com/acpopescu/bitsandbytes/tree/cmake_windows Still WIP.

To Deploy copy build/Release/*.* to ./bitsandbytes/

For reference and diff - https://github.com/TimDettmers/bitsandbytes/pull/229

[acpopescu]

Automated tests look promising on 11.7, at least on GPU :) https://github.com/TimDettmers/bitsandbytes/pull/229#issuecomment-1486142982

python -m build was able to build a wheel file and that worked with 11.7: CPU is NOT TESTED below: bitsandbytes-0.37.2-py3-none-any.whl.zip