text-generation-webui
text-generation-webui copied to clipboard
oobabooga
Windows bitsandbytes error
Describe the bug
Used the 1 click installer.
Is there an existing issue for this?
- [X] I have searched the existing issues
Reproduction
Use the 1 click installer for Windows. Run start-webui.bat
Screenshot
No response
Logs
Starting the web UI...
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
D:\oobabooga\installer_files\env\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('D')}
warn(msg)
D:\oobabooga\installer_files\env\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: D:\oobabooga\installer_files\env did not contain libcudart.so as expected! Searching further paths...
warn(msg)
D:\oobabooga\installer_files\env\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/oobabooga/installer_files/env/etc/xml/catalog'), WindowsPath('/D'), WindowsPath('file')}
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
D:\oobabooga\installer_files\env\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
warn(msg)
CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed? If you are on a cluster, make sure you are on a CUDA machine!
D:\oobabooga\installer_files\env\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
warn(msg)
D:\oobabooga\installer_files\env\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: No GPU detected! Check your CUDA paths. Proceeding to load CPU-only library...
warn(msg)
CUDA SETUP: Loading binary D:\oobabooga\installer_files\env\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so...
argument of type 'WindowsPath' is not iterable
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed? If you are on a cluster, make sure you are on a CUDA machine!
CUDA SETUP: Loading binary D:\oobabooga\installer_files\env\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so...
argument of type 'WindowsPath' is not iterable
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed? If you are on a cluster, make sure you are on a CUDA machine!
CUDA SETUP: Loading binary D:\oobabooga\installer_files\env\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so...
argument of type 'WindowsPath' is not iterable
CUDA SETUP: Problem: The main issue seems to be that the main CUDA library was not detected.
CUDA SETUP: Solution 1): Your paths are probably not up-to-date. You can update them via: sudo ldconfig.
CUDA SETUP: Solution 2): If you do not have sudo rights, you can do the following:
CUDA SETUP: Solution 2a): Find the cuda library via: find / -name libcuda.so 2>/dev/null
CUDA SETUP: Solution 2b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_2a
CUDA SETUP: Solution 2c): For a permanent solution add the export from 2b into your .bashrc file, located at ~/.bashrc
Traceback (most recent call last):
File "D:\oobabooga\text-generation-webui\server.py", line 18, in <module>
from modules.LoRA import add_lora_to_model
File "D:\oobabooga\text-generation-webui\modules\LoRA.py", line 3, in <module>
from peft import PeftModel
File "D:\oobabooga\installer_files\env\lib\site-packages\peft\__init__.py", line 22, in <module>
from .mapping import MODEL_TYPE_TO_PEFT_MODEL_MAPPING, PEFT_TYPE_TO_CONFIG_MAPPING, get_peft_config, get_peft_model
File "D:\oobabooga\installer_files\env\lib\site-packages\peft\mapping.py", line 16, in <module>
from .peft_model import (
File "D:\oobabooga\installer_files\env\lib\site-packages\peft\peft_model.py", line 31, in <module>
from .tuners import LoraModel, PrefixEncoder, PromptEmbedding, PromptEncoder
File "D:\oobabooga\installer_files\env\lib\site-packages\peft\tuners\__init__.py", line 20, in <module>
from .lora import LoraConfig, LoraModel
File "D:\oobabooga\installer_files\env\lib\site-packages\peft\tuners\lora.py", line 36, in <module>
import bitsandbytes as bnb
File "D:\oobabooga\installer_files\env\lib\site-packages\bitsandbytes\__init__.py", line 7, in <module>
from .autograd._functions import (
File "D:\oobabooga\installer_files\env\lib\site-packages\bitsandbytes\autograd\__init__.py", line 1, in <module>
from ._functions import undo_layout, get_inverse_transform_indices
File "D:\oobabooga\installer_files\env\lib\site-packages\bitsandbytes\autograd\_functions.py", line 9, in <module>
import bitsandbytes.functional as F
File "D:\oobabooga\installer_files\env\lib\site-packages\bitsandbytes\functional.py", line 17, in <module>
from .cextension import COMPILED_WITH_CUDA, lib
File "D:\oobabooga\installer_files\env\lib\site-packages\bitsandbytes\cextension.py", line 22, in <module>
raise RuntimeError('''
RuntimeError:
CUDA Setup failed despite GPU being available. Inspect the CUDA SETUP outputs above to fix your environment!
If you cannot find any issues and suspect a bug, please open an issue with detals about your environment:
https://github.com/TimDettmers/bitsandbytes/issues
Press any key to continue . . .
System Info
Windows 11
3090
CUDA 11.8 and CUDA 11.6
Have you done this? https://github.com/oobabooga/text-generation-webui/issues/173#issuecomment-1456087035
There's a link to another comment in that post https://github.com/oobabooga/text-generation-webui/issues/147#issuecomment-1456040134
Something appears to have changed. I can no longer get bitsandbytes to function after doing a git pull, and then a full reinstall. Patching bitsandbytes no longer appears to work.
C:\Users\user\miniconda3\envs\textgen\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)! warn(msg) C:\Users\user\miniconda3\envs\textgen\lib\site-packages\bitsandbytes\cuda_setup\main.py:136: UserWarning: WARNING: No GPU detected! Check your CUDA paths. Proceeding to load CPU-only library... warn(msg) CUDA SETUP: Loading binary C:\Users\user\miniconda3\envs\textgen\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so... [WinError 193] %1 is not a valid Win32 application CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64... CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed? If you are on a cluster, make sure you are on a CUDA machine! CUDA SETUP: Loading binary C:\Users\user\miniconda3\envs\textgen\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so... [WinError 193] %1 is not a valid Win32 application C:\Users\user\miniconda3\envs\textgen\lib\site-packages\bitsandbytes\cextension.py:31: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable. warn("The installed version of bitsandbytes was compiled without GPU support. " Loading llama-7b-hf... Warning: torch.cuda.is_available() returned False. This means that no GPU has been detected. Falling back to CPU mode.
I've just faced with literally the same error stream, by updating via one-click installer right now. All I know that previous successful update was done 17-03 in the early morning. Since the update the Web UI is no longer operable anyway.
Copying the dll and modifying main.py as in this comment https://github.com/oobabooga/text-generation-webui/issues/147#issuecomment-1456040134 got past the bitsandbytes error, but now; llama-7b-hf model loads the model shards but then errors out with "ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported." opt-350m works.
For the LLaMATokenizer issue you need a different transformers, ie pip uninstall -y transformers pip install git+https://github.com/zphang/transformers.git@68d640f7c368bcaaaecfc678f11908ebbd3d6176 That gets llama-7b-hf loading and working again.
I've finally dealt with an error of corrupted WindowsPath by manually updating both the WebUI and the one-click scripts as well. Now the .bat files must be launched strictly from the Git command prompt as an Administrator.
Can we close this issue? It looks like the problems are resolved with earlier fixes?
It started with the latest update yesterday. If you close it now every person using the one-click installer is going to have these same problems.
@SoftologyPro The issue template literally asks you to search before posting, so even if you close the issue people will be able to find it.
But what about the problem? It started just yesterday. Can it be rolled back to the working version? If it is not fixed every Windows user who uses the one-click installer will have to manually make the above changes.
@SoftologyPro I had this problem and fixed it. I wrote some installation instructions over here for my Discord bot if you want to try them.
@xNul I have fixes above. I was hoping the dev(s) would fix the git so I don't have to patch it every update.
Yeah, I'm getting it now too, very frustrating However, this temporary fix worked for me: https://github.com/oobabooga/text-generation-webui/issues/20#issuecomment-1411650652
8-bit should work out of the box with the new one-click installer
https://github.com/oobabooga/text-generation-webui#one-click-installers
sed does not work by default in Windows so one click installer fails.
The one-click-installer installs Git which comes with a Windows-compatible version of sed. When using the installer, did the console show an error saying that it couldn't find sed?
@jllllll
did the console show an error saying that it couldn't find sed?
Yup. EDIT: OK, not an error that "it couldn't find sed", but sed-related. At least, for me. Tried both options A and B, for Nvidia and "plain" CPU. But it's the last error before the installer fails: sed: can't read c:\MyFolder\oobabooga-windows\installer_files\env\lib\site-packages\bitsandbytes\cuda_setup\main.py: Not a directory
Before that, I get another error:
Could not install packages due to an OSError: [Errno 2] No such file or directory: 'C:\MyFolder\oobabooga-windows\Installer_files\env\Lib\site-packages\bitsandbytes\init.py'
What's strange is that in both the CUDA and non-CUDA ("plain" CPU) setup it seems like the script's failing when sed tries to access .. .. .. ..\cuda_setup\main.py with a "Not a directory" message.
Unfortunately, I currently don't have access to a local Linux installation to try it on. The above's on Windows 11 Pro 64bit with an RTX 3070, Ryzen 5900x, 32GBs RAM.
PS: I've tried it with CMD and PowerShell, "normal" and "elevated". Am I doing something wrong?
@jllllll
did the console show an error saying that it couldn't find sed?
Yup. EDIT: OK, not an error that "it couldn't find sed", but sed-related. At least, for me. Tried both options A and B, for Nvidia and "plain" CPU. But it's the last error before the installer fails: sed: can't read c:\MyFolder\oobabooga-windows\installer_files\env\lib\site-packages\bitsandbytes\cuda_setup\main.py: Not a directory
Before that, I get another error:
Could not install packages due to an OSError: [Errno 2] No such file or directory: 'C:\MyFolder\oobabooga-windows\Installer_files\env\Lib\site-packages\bitsandbytes*init*.py'
What's strange is that in both the CUDA and non-CUDA ("plain" CPU) setup it seems like the script's failing when sed tries to access .. .. .. ..\cuda_setup\main.py with a "Not a directory" message.
Unfortunately, I currently don't have access to a local Linux installation to try it on. The above's on Windows 11 Pro 64bit with an RTX 3070, Ryzen 5900x, 32GBs RAM.
PS: I've tried it with CMD and PowerShell, "normal" and "elevated". Am I doing something wrong?
Honestly? I've no idea. I'm currently setting up a proper bitsandbytes Windows compilation. Once I'm done, I'll need someone to test it on a RTX card as I don't have one. I've already confirmed that it works on my 1080ti. This will eliminate the need for any hacky edits with sed.
For anyone with an RTX card:
Here is the wheel file: bitsandbytes-0.37.2-py3-none-any.whl.zip
Extract it next to micromamba-cmd.bat, run that .bat and enter the following commands in order:
python -m pip uninstall bitsandbytes
python -m pip install bitsandbytes-0.37.2-py3-none-any.whl
exit
If you didn't use the new one-click-installer, then just run those commands within the environment you are using.
After that, try to run a model in 8-bit and see if it works.
@jllllll I've downloaded and extracted the wheel file you provided (not that I know what "a wheel file" is :-D ), ran micromamba-cmd.bat, and entered the two pip commands that, AFAIU, are meant to uninstall any installed instance of a package called bitsandbytes and install the version "from the wheel file" (or something along those lines)... Right?
- No bitsandbytes package was found already installed. Maybe that's a hint on where the "one-click-installer" fails?
- ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: 'C:\MyFolder\oobabooga-windows\installer_files\env\Lib\site-packages\bitsandbytes\init.py'
As old text adventures famously put it, "what now?" :-)
Sorry, I only know the basics of Python, JS, Node, and although I can make sense of both the Windows and the Linux file system, when "they're mixed" I get a bit lost (in projects like this, that basically require the installation of extra packages neither in a System or Program Files folder, as is the norm on Windows, nor in the user's Home dir, or Var/Bin/Sbin, as I'm familiar with from Linux). Maybe I'm missing something?
@jllllll I've downloaded and extracted the wheel file you provided (not that I know what "a wheel file" is :-D ), ran micromamba-cmd.bat, and entered the two pip commands that, AFAIU, are meant to uninstall any installed instance of a package called bitsandbytes and install the version "from the wheel file" (or something along those lines)... Right?
1. No bitsandbytes package was found already installed. Maybe that's a hint on where the "one-click-installer" fails? 2. ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: 'C:\MyFolder\oobabooga-windows\installer_files\env\Lib\site-packages\bitsandbytes\**init**.py'As old text adventures famously put it, "what now?" :-)
Sorry, I only know the basics of Python, JS, Node, and although I can make sense of both the Windows and the Linux file system, when "they're mixed" I get a bit lost (in projects like this, that basically require the installation of extra packages neither in a System or Program Files folder, as is the norm on Windows, nor in the user's Home dir, or Var/Bin/Sbin, as I'm familiar with from Linux). Maybe I'm missing something?
Only thing I can think of is trying this version of the installer: https://github.com/jllllll/one-click-installers/tree/oobabooga-windows-miniconda
It uses a very different method that hopefully should work.
@jllllll Thanks, that worked without the bitsandbytes error.
However, it borked due to the directory error... until... Er... I manually deleted all files from the previous installation.
Yeah, I initially tried replacing the old files with the alternative installer files you provided. It didn't work, and I guess it was because of some Windows Access Rights-related issue with one of the folders created by the previous installer.
So, if this fails for anyone else in a similar way, it's worth deleting the existing files, and starting from scratch.
I use an alternate installer but I've edited my start-webui.bat to work around the issue:
call pip install bitsandbytes-windows call python server.py --auto-devices --chat --model-menu --load-in-8bit
This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}