text-generation-webui
text-generation-webui copied to clipboard
oobabooga
Cuda extension not installed
Describe the bug
Please help I am tired of this bug I am trying to fix it for a week now I tried lots of think non work so far it did't even change the error
Is there an existing issue for this?
- [X] I have searched the existing issues
Reproduction
I dont know
Screenshot
No response
Logs
Starting the web UI...
D:\Gpt\oobabooga-windows\installer_files\env\lib\site-packages\numpy\_distributor_init.py:30: UserWarning: loaded more than 1 DLL from .libs:
D:\Gpt\oobabooga-windows\installer_files\env\lib\site-packages\numpy\.libs\libopenblas.FB5AE2TYXYH2IJRDKGDGQ3XBKLKTF43H.gfortran-win_amd64.dll
D:\Gpt\oobabooga-windows\installer_files\env\lib\site-packages\numpy\.libs\libopenblas64__v0.3.21-gcc_10_3_0.dll
warnings.warn("loaded more than 1 DLL from .libs:"
D:\Gpt\oobabooga-windows\installer_files\env\lib\site-packages\bitsandbytes\cextension.py:31: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable.
warn("The installed version of bitsandbytes was compiled without GPU support. "
Loading anon8231489123_vicuna-13b-GPTQ-4bit-128g...
CUDA extension not installed.
Found the following quantized model: models\anon8231489123_vicuna-13b-GPTQ-4bit-128g\vicuna-13b-4bit-128g.safetensors
Loading model ...
Press any key to continue . . .
System Info
3080 10 gb i7 7700 16gb ram
Have you tried fully re-installing using the newest installers? ( fully deleting text-generation-webui and installer_files )
https://github.com/jllllll/one-click-installers https://github.com/xNul/one-click-installers
@PrensesCemal Did you happen to change anything in the installer such as the Python version? Other than that, I have no clue as to what could be causing this as I am unable to reproduce it on my system.
I have the same problem
Gradio HTTP request redirected to localhost :)
bin C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.dll
C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\bitsandbytes\cextension.py:33: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
warn("The installed version of bitsandbytes was compiled without GPU support. "
Loading anon8231489123_vicuna-13b-GPTQ-4bit-128g...
CUDA extension not installed.
Found the following quantized model: models\anon8231489123_vicuna-13b-GPTQ-4bit-128g\vicuna-13b-4bit-128g.safetensors
Loading model ...
Done.
Traceback (most recent call last):
File "C:\AI\oobabooga_windows\text-generation-webui\server.py", line 921, in
Done! Press any key to continue . . .
@TheSigma13 Open the cmd script and enter this command:
python -m torch.utils.collect_env
If it says that you have the cpu version of torch, enter this:
python -m pip install https://download.pytorch.org/whl/cu117/torch-2.0.0%2Bcu117-cp310-cp310-win_amd64.whl --force-reinstall
The CUDA extension can be manually installed with this:
python -m pip install https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl --force-reinstall
Thank you, I did both force install commands that You mentioned but still get the exact same error message
C:\Windows\system32>python -m pip install https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl --force-reinstall Collecting quant-cuda==0.0.0 Using cached https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl (398 kB) Installing collected packages: quant-cuda Attempting uninstall: quant-cuda Found existing installation: quant-cuda 0.0.0 Uninstalling quant-cuda-0.0.0: Successfully uninstalled quant-cuda-0.0.0 Successfully installed quant-cuda-0.0.0
It seems that Cuda extension is installed but the oobabooga can't find it for some reason
cuda fails when installing ownloading and Extracting Packages
**CondaHTTPError: HTTP 000 CONNECTION FAILED for url https://conda.anaconda.org/pytorch/win-64/pytorch-2.0.0-py3.10_cuda11.7_cudnn8_0.tar.bz2 Elapsed: -
An HTTP error occurred when trying to retrieve this URL. HTTP errors are often intermittent, and a simple retry will get you on your way.**
and when retrying it downloads all files again and then same error
`What is your GPU
A) NVIDIA
B) AMD
C) Apple M Series
D) None (I want to run in CPU mode)
Input> A
Collecting package metadata (current_repodata.json): done
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 23.1.0
latest version: 23.3.1
Please update conda by running
$ conda update -n base -c defaults conda
Or to minimize the number of packages updated during conda update use
conda install conda=23.3.1
## Package Plan ##
environment location: X:\Auto-TEXT-WEBUI\gpt\installer_files\env
added / updated specs:
- cuda-toolkit
- git
- ninja
- pytorch-cuda=11.7
- pytorch==2[build=py3.10_cuda11.7*]
- torchaudio
- torchvision
The following packages will be downloaded:
package | build
---------------------------|-----------------
cuda-compiler-11.7.0 | 0 1 KB nvidia/label/cuda-11.7.0
cuda-memcheck-11.7.50 | 0 183 KB nvidia/label/cuda-11.7.0
cuda-nvprof-11.7.50 | 0 1.5 MB nvidia/label/cuda-11.7.0
cuda-nvprune-11.7.50 | 0 152 KB nvidia/label/cuda-11.7.0
giflib-5.2.1 | h8cc25b3_3 88 KB
libcublas-dev-11.10.1.25 | 0 286.3 MB nvidia/label/cuda-11.7.0
libcurand-dev-10.2.10.50 | 0 49.1 MB nvidia/label/cuda-11.7.0
libpng-1.6.39 | h8cc25b3_0 369 KB
pytorch-2.0.0 |py3.10_cuda11.7_cudnn8_0 1.17 GB pytorch
------------------------------------------------------------
Total: 1.50 GB
The following NEW packages will be INSTALLED:
blas pkgs/main/win-64::blas-1.0-mkl
brotlipy pkgs/main/win-64::brotlipy-0.7.0-py310h2bbff1b_1002
certifi pkgs/main/win-64::certifi-2022.12.7-py310haa95532_0
cffi pkgs/main/win-64::cffi-1.15.1-py310h2bbff1b_3
charset-normalizer pkgs/main/noarch::charset-normalizer-2.0.4-pyhd3eb1b0_0
cryptography pkgs/main/win-64::cryptography-39.0.1-py310h21b164f_0
cuda-cccl nvidia/label/cuda-11.7.0/win-64::cuda-cccl-11.7.58-0
cuda-command-line~ nvidia/label/cuda-11.7.0/win-64::cuda-command-line-tools-11.7.0-0
cuda-compiler nvidia/label/cuda-11.7.0/win-64::cuda-compiler-11.7.0-0
cuda-cudart nvidia/label/cuda-11.7.0/win-64::cuda-cudart-11.7.60-0
cuda-cudart-dev nvidia/label/cuda-11.7.0/win-64::cuda-cudart-dev-11.7.60-0
cuda-cuobjdump nvidia/label/cuda-11.7.0/win-64::cuda-cuobjdump-11.7.50-0
cuda-cupti nvidia/label/cuda-11.7.0/win-64::cuda-cupti-11.7.50-0
cuda-cuxxfilt nvidia/label/cuda-11.7.0/win-64::cuda-cuxxfilt-11.7.50-0
cuda-documentation nvidia/label/cuda-11.7.0/win-64::cuda-documentation-11.7.50-0
cuda-libraries nvidia/label/cuda-11.7.0/win-64::cuda-libraries-11.7.0-0
cuda-libraries-dev nvidia/label/cuda-11.7.0/win-64::cuda-libraries-dev-11.7.0-0
cuda-memcheck nvidia/label/cuda-11.7.0/win-64::cuda-memcheck-11.7.50-0
cuda-nsight-compu~ nvidia/label/cuda-11.7.0/win-64::cuda-nsight-compute-11.7.0-0
cuda-nvcc nvidia/label/cuda-11.7.0/win-64::cuda-nvcc-11.7.64-0
cuda-nvdisasm nvidia/label/cuda-11.7.0/win-64::cuda-nvdisasm-11.7.50-0
cuda-nvml-dev nvidia/label/cuda-11.7.0/win-64::cuda-nvml-dev-11.7.50-0
cuda-nvprof nvidia/label/cuda-11.7.0/win-64::cuda-nvprof-11.7.50-0
cuda-nvprune nvidia/label/cuda-11.7.0/win-64::cuda-nvprune-11.7.50-0
cuda-nvrtc nvidia/label/cuda-11.7.0/win-64::cuda-nvrtc-11.7.50-0
cuda-nvrtc-dev nvidia/label/cuda-11.7.0/win-64::cuda-nvrtc-dev-11.7.50-0
cuda-nvtx nvidia/label/cuda-11.7.0/win-64::cuda-nvtx-11.7.50-0
cuda-nvvp nvidia/label/cuda-11.7.0/win-64::cuda-nvvp-11.7.50-0
cuda-runtime nvidia/label/cuda-11.7.0/win-64::cuda-runtime-11.7.0-0
cuda-sanitizer-api nvidia/label/cuda-11.7.0/win-64::cuda-sanitizer-api-11.7.50-0
cuda-toolkit nvidia/label/cuda-11.7.0/win-64::cuda-toolkit-11.7.0-0
cuda-tools nvidia/label/cuda-11.7.0/win-64::cuda-tools-11.7.0-0
cuda-visual-tools nvidia/label/cuda-11.7.0/win-64::cuda-visual-tools-11.7.0-0
filelock pkgs/main/win-64::filelock-3.9.0-py310haa95532_0
freetype pkgs/main/win-64::freetype-2.12.1-ha860e81_0
giflib pkgs/main/win-64::giflib-5.2.1-h8cc25b3_3
git pkgs/main/win-64::git-2.34.1-haa95532_0
idna pkgs/main/win-64::idna-3.4-py310haa95532_0
intel-openmp pkgs/main/win-64::intel-openmp-2021.4.0-haa95532_3556
jinja2 pkgs/main/win-64::jinja2-3.1.2-py310haa95532_0
jpeg pkgs/main/win-64::jpeg-9e-h2bbff1b_1
lerc pkgs/main/win-64::lerc-3.0-hd77b12b_0
libcublas nvidia/label/cuda-11.7.0/win-64::libcublas-11.10.1.25-0
libcublas-dev nvidia/label/cuda-11.7.0/win-64::libcublas-dev-11.10.1.25-0
libcufft nvidia/label/cuda-11.7.0/win-64::libcufft-10.7.2.50-0
libcufft-dev nvidia/label/cuda-11.7.0/win-64::libcufft-dev-10.7.2.50-0
libcurand nvidia/label/cuda-11.7.0/win-64::libcurand-10.2.10.50-0
libcurand-dev nvidia/label/cuda-11.7.0/win-64::libcurand-dev-10.2.10.50-0
libcusolver nvidia/label/cuda-11.7.0/win-64::libcusolver-11.3.5.50-0
libcusolver-dev nvidia/label/cuda-11.7.0/win-64::libcusolver-dev-11.3.5.50-0
libcusparse nvidia/label/cuda-11.7.0/win-64::libcusparse-11.7.3.50-0
libcusparse-dev nvidia/label/cuda-11.7.0/win-64::libcusparse-dev-11.7.3.50-0
libdeflate pkgs/main/win-64::libdeflate-1.17-h2bbff1b_0
libnpp nvidia/label/cuda-11.7.0/win-64::libnpp-11.7.3.21-0
libnpp-dev nvidia/label/cuda-11.7.0/win-64::libnpp-dev-11.7.3.21-0
libnvjpeg nvidia/label/cuda-11.7.0/win-64::libnvjpeg-11.7.2.34-0
libnvjpeg-dev nvidia/label/cuda-11.7.0/win-64::libnvjpeg-dev-11.7.2.34-0
libpng pkgs/main/win-64::libpng-1.6.39-h8cc25b3_0
libtiff pkgs/main/win-64::libtiff-4.5.0-h6c2663c_2
libuv pkgs/main/win-64::libuv-1.44.2-h2bbff1b_0
libwebp pkgs/main/win-64::libwebp-1.2.4-hbc33d0d_1
libwebp-base pkgs/main/win-64::libwebp-base-1.2.4-h2bbff1b_1
lz4-c pkgs/main/win-64::lz4-c-1.9.4-h2bbff1b_0
markupsafe pkgs/main/win-64::markupsafe-2.1.1-py310h2bbff1b_0
mkl pkgs/main/win-64::mkl-2021.4.0-haa95532_640
mkl-service pkgs/main/win-64::mkl-service-2.4.0-py310h2bbff1b_0
mkl_fft pkgs/main/win-64::mkl_fft-1.3.1-py310ha0764ea_0
mkl_random pkgs/main/win-64::mkl_random-1.2.2-py310h4ed8f06_0
mpmath pkgs/main/win-64::mpmath-1.2.1-py310haa95532_0
networkx pkgs/main/win-64::networkx-2.8.4-py310haa95532_1
ninja pkgs/main/win-64::ninja-1.10.2-haa95532_5
ninja-base pkgs/main/win-64::ninja-base-1.10.2-h6d14046_5
nsight-compute nvidia/label/cuda-11.7.0/win-64::nsight-compute-2022.2.0.13-0
numpy pkgs/main/win-64::numpy-1.24.3-py310hdc03b94_0
numpy-base pkgs/main/win-64::numpy-base-1.24.3-py310h3caf3d7_0
pillow pkgs/main/win-64::pillow-9.4.0-py310hd77b12b_0
pycparser pkgs/main/noarch::pycparser-2.21-pyhd3eb1b0_0
pyopenssl pkgs/main/win-64::pyopenssl-23.0.0-py310haa95532_0
pysocks pkgs/main/win-64::pysocks-1.7.1-py310haa95532_0
pytorch pytorch/win-64::pytorch-2.0.0-py3.10_cuda11.7_cudnn8_0
pytorch-cuda pytorch/win-64::pytorch-cuda-11.7-h16d0643_3
pytorch-mutex pytorch/noarch::pytorch-mutex-1.0-cuda
requests pkgs/main/win-64::requests-2.29.0-py310haa95532_0
six pkgs/main/noarch::six-1.16.0-pyhd3eb1b0_1
sympy pkgs/main/win-64::sympy-1.11.1-py310haa95532_0
torchaudio pytorch/win-64::torchaudio-2.0.0-py310_cu117
torchvision pytorch/win-64::torchvision-0.15.0-py310_cu117
typing_extensions pkgs/main/win-64::typing_extensions-4.5.0-py310haa95532_0
urllib3 pkgs/main/win-64::urllib3-1.26.15-py310haa95532_0
win_inet_pton pkgs/main/win-64::win_inet_pton-1.1.0-py310haa95532_0
zstd pkgs/main/win-64::zstd-1.5.5-hd43e919_0
Downloading and Extracting Packages
**CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/pytorch/win-64/pytorch-2.0.0-py3.10_cuda11.7_cudnn8_0.tar.bz2>
Elapsed: -
An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.**
Cloning into 'text-generation-webui'...
remote: Enumerating objects: 6369, done.
remote: Counting objects: 100% (6369/6369), done.
remote: Compressing objects: 100% (2136/2136), done.
remote: Total 6369 (delta 4261), reused 6137 (delta 4123), pack-reused 0
Receiving objects: 100% (6369/6369), 2.31 MiB | 2.52 MiB/s, done.
Resolving deltas: 100% (4261/4261), done.
Collecting bitsandbytes==0.38.1
Using cached https://github.com/jllllll/bitsandbytes-windows-webui/raw/main/bitsandbytes-0.38.1-py3-none-any.whl (82.6 MB)
[notice] A new release of pip available: 22.3 -> 23.1.2
[notice] To update, run: python.exe -m pip install --upgrade pip
Already up to date.
Collecting git+https://github.com/huggingface/peft (from -r requirements.txt (line 16))
Cloning https://github.com/huggingface/peft to c:\users\hassan\appdata\local\temp\pip-req-build-8tpfmgnc
Running command git clone --filter=blob:none --quiet https://github.com/huggingface/peft 'C:\Users\Hassan\AppData\Local\Temp\pip-req-build-8tpfmgnc'
Resolved https://github.com/huggingface/peft to commit 632997d1fb776c3cf05d8c2537ac9a98a7ce9435
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Ignoring bitsandbytes: markers 'platform_system != "Windows"' don't match your environment
Ignoring llama-cpp-python: markers 'platform_system != "Windows"' don't match your environment
Collecting llama-cpp-python==0.1.36
Downloading https://github.com/abetlen/llama-cpp-python/releases/download/v0.1.36/llama_cpp_python-0.1.36-cp310-cp310-win_amd64.whl (128 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 128.7/128.7 kB 420.2 kB/s eta 0:00:00
Collecting accelerate==0.18.0
Using cached accelerate-0.18.0-py3-none-any.whl (215 kB)
Collecting colorama
Using cached colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Collecting datasets
Using cached datasets-2.12.0-py3-none-any.whl (474 kB)
Collecting flexgen==0.1.7
Using cached flexgen-0.1.7-py3-none-any.whl (50 kB)
Collecting gradio==3.25.0
Using cached gradio-3.25.0-py3-none-any.whl (17.5 MB)
Collecting markdown
Using cached Markdown-3.4.3-py3-none-any.whl (93 kB)
Collecting numpy
Using cached numpy-1.24.3-cp310-cp310-win_amd64.whl (14.8 MB)
Collecting pandas
Using cached pandas-2.0.1-cp310-cp310-win_amd64.whl (10.7 MB)
Collecting Pillow>=9.5.0
Using cached Pillow-9.5.0-cp310-cp310-win_amd64.whl (2.5 MB)
Collecting pyyaml
Using cached PyYAML-6.0-cp310-cp310-win_amd64.whl (151 kB)
Collecting requests
Using cached requests-2.29.0-py3-none-any.whl (62 kB)
Collecting rwkv==0.7.3
Using cached rwkv-0.7.3-py3-none-any.whl (16 kB)
Collecting safetensors==0.3.0
Using cached safetensors-0.3.0-cp310-cp310-win_amd64.whl (262 kB)
Collecting sentencepiece
Using cached sentencepiece-0.1.98-cp310-cp310-win_amd64.whl (977 kB)
Collecting tqdm
Using cached tqdm-4.65.0-py3-none-any.whl (77 kB)
Collecting transformers==4.28.1
Using cached transformers-4.28.1-py3-none-any.whl (7.0 MB)
Collecting packaging>=20.0
Using cached packaging-23.1-py3-none-any.whl (48 kB)
Collecting psutil
Using cached psutil-5.9.5-cp36-abi3-win_amd64.whl (255 kB)
Collecting torch>=1.4.0
Using cached torch-2.0.0-cp310-cp310-win_amd64.whl (172.3 MB)
Collecting attrs
Using cached attrs-23.1.0-py3-none-any.whl (61 kB)
Collecting pulp
Using cached PuLP-2.7.0-py3-none-any.whl (14.3 MB)
Collecting huggingface-hub>=0.13.0
Using cached huggingface_hub-0.14.1-py3-none-any.whl (224 kB)
Collecting semantic-version
Using cached semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)
Collecting orjson
Using cached orjson-3.8.11-cp310-none-win_amd64.whl (196 kB)
Collecting pydantic
Using cached pydantic-1.10.7-cp310-cp310-win_amd64.whl (2.1 MB)
Collecting matplotlib
Using cached matplotlib-3.7.1-cp310-cp310-win_amd64.whl (7.6 MB)
Collecting ffmpy
Using cached ffmpy-0.3.0-py3-none-any.whl
Collecting pydub
Using cached pydub-0.25.1-py2.py3-none-any.whl (32 kB)
Collecting altair>=4.2.0
Using cached altair-4.2.2-py3-none-any.whl (813 kB)
Collecting python-multipart
Using cached python_multipart-0.0.6-py3-none-any.whl (45 kB)
Collecting jinja2
Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting uvicorn
Using cached uvicorn-0.22.0-py3-none-any.whl (58 kB)
Collecting aiohttp
Using cached aiohttp-3.8.4-cp310-cp310-win_amd64.whl (319 kB)
Collecting markupsafe
Using cached MarkupSafe-2.1.2-cp310-cp310-win_amd64.whl (16 kB)
Collecting fastapi
Using cached fastapi-0.95.1-py3-none-any.whl (56 kB)
Collecting gradio-client>=0.0.8
Using cached gradio_client-0.1.4-py3-none-any.whl (286 kB)
Collecting mdit-py-plugins<=0.3.3
Using cached mdit_py_plugins-0.3.3-py3-none-any.whl (50 kB)
Collecting httpx
Using cached httpx-0.24.0-py3-none-any.whl (75 kB)
Collecting markdown-it-py[linkify]>=2.0.0
Using cached markdown_it_py-2.2.0-py3-none-any.whl (84 kB)
Collecting websockets>=10.0
Using cached websockets-11.0.2-cp310-cp310-win_amd64.whl (124 kB)
Collecting aiofiles
Using cached aiofiles-23.1.0-py3-none-any.whl (14 kB)
Collecting typing-extensions
Using cached typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Collecting tokenizers>=0.13.2
Using cached tokenizers-0.13.3-cp310-cp310-win_amd64.whl (3.5 MB)
Collecting filelock
Using cached filelock-3.12.0-py3-none-any.whl (10 kB)
Collecting regex!=2019.12.17
Using cached regex-2023.3.23-cp310-cp310-win_amd64.whl (267 kB)
Collecting multiprocess
Using cached multiprocess-0.70.14-py310-none-any.whl (134 kB)
Collecting xxhash
Using cached xxhash-3.2.0-cp310-cp310-win_amd64.whl (30 kB)
Collecting responses<0.19
Using cached responses-0.18.0-py3-none-any.whl (38 kB)
Collecting dill<0.3.7,>=0.3.0
Using cached dill-0.3.6-py3-none-any.whl (110 kB)
Collecting pyarrow>=8.0.0
Using cached pyarrow-11.0.0-cp310-cp310-win_amd64.whl (20.6 MB)
Collecting fsspec[http]>=2021.11.1
Using cached fsspec-2023.4.0-py3-none-any.whl (153 kB)
Collecting tzdata>=2022.1
Using cached tzdata-2023.3-py2.py3-none-any.whl (341 kB)
Collecting pytz>=2020.1
Using cached pytz-2023.3-py2.py3-none-any.whl (502 kB)
Collecting python-dateutil>=2.8.2
Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
Collecting idna<4,>=2.5
Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting charset-normalizer<4,>=2
Using cached charset_normalizer-3.1.0-cp310-cp310-win_amd64.whl (97 kB)
Collecting entrypoints
Using cached entrypoints-0.4-py3-none-any.whl (5.3 kB)
Collecting toolz
Using cached toolz-0.12.0-py3-none-any.whl (55 kB)
Collecting jsonschema>=3.0
Using cached jsonschema-4.17.3-py3-none-any.whl (90 kB)
Collecting async-timeout<5.0,>=4.0.0a3
Using cached async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
Collecting frozenlist>=1.1.1
Using cached frozenlist-1.3.3-cp310-cp310-win_amd64.whl (33 kB)
Collecting aiosignal>=1.1.2
Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting multidict<7.0,>=4.5
Using cached multidict-6.0.4-cp310-cp310-win_amd64.whl (28 kB)
Collecting yarl<2.0,>=1.0
Using cached yarl-1.9.2-cp310-cp310-win_amd64.whl (61 kB)
Collecting mdurl~=0.1
Using cached mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Collecting linkify-it-py<3,>=1
Using cached linkify_it_py-2.0.0-py3-none-any.whl (19 kB)
Collecting six>=1.5
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting networkx
Using cached networkx-3.1-py3-none-any.whl (2.1 MB)
Collecting sympy
Using cached sympy-1.11.1-py3-none-any.whl (6.5 MB)
Collecting starlette<0.27.0,>=0.26.1
Using cached starlette-0.26.1-py3-none-any.whl (66 kB)
Collecting httpcore<0.18.0,>=0.15.0
Using cached httpcore-0.17.0-py3-none-any.whl (70 kB)
Collecting sniffio
Using cached sniffio-1.3.0-py3-none-any.whl (10 kB)
Collecting fonttools>=4.22.0
Using cached fonttools-4.39.3-py3-none-any.whl (1.0 MB)
Collecting contourpy>=1.0.1
Using cached contourpy-1.0.7-cp310-cp310-win_amd64.whl (162 kB)
Collecting pyparsing>=2.3.1
Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting kiwisolver>=1.0.1
Using cached kiwisolver-1.4.4-cp310-cp310-win_amd64.whl (55 kB)
Collecting cycler>=0.10
Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting h11>=0.8
Using cached h11-0.14.0-py3-none-any.whl (58 kB)
Collecting click>=7.0
Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting anyio<5.0,>=3.0
Using cached anyio-3.6.2-py3-none-any.whl (80 kB)
Collecting pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0
Using cached pyrsistent-0.19.3-cp310-cp310-win_amd64.whl (62 kB)
Collecting uc-micro-py
Using cached uc_micro_py-1.0.1-py3-none-any.whl (6.2 kB)
Collecting mpmath>=0.19
Using cached mpmath-1.3.0-py3-none-any.whl (536 kB)
Building wheels for collected packages: peft
Building wheel for peft (pyproject.toml) ... done
Created wheel for peft: filename=peft-0.3.0.dev0-py3-none-any.whl size=55901 sha256=ab4fa9a208523d1d12b260c7a0c8d01f70ba0658d7c622332b61b1e7cf2f3d0c
Stored in directory: C:\Users\Hassan\AppData\Local\Temp\pip-ephem-wheel-cache-uyayppm1\wheels\4c\16\67\1002a2d4daa822eff130e6d85b90051b75d2ce0d26b9448e4a
Successfully built peft
Installing collected packages: tokenizers, sentencepiece, safetensors, pytz, pydub, pulp, mpmath, ffmpy, xxhash, websockets, urllib3, uc-micro-py, tzdata, typing-extensions, toolz, sympy, sniffio, six, semantic-version, rwkv, regex, pyyaml, python-multipart, pyrsistent, pyparsing, psutil, Pillow, packaging, orjson, numpy, networkx, multidict, mdurl, markupsafe, markdown, kiwisolver, idna, h11, fsspec, frozenlist, fonttools, filelock, entrypoints, dill, cycler, colorama, charset-normalizer, certifi, attrs, async-timeout, aiofiles, yarl, tqdm, requests, python-dateutil, pydantic, pyarrow, multiprocess, markdown-it-py, llama-cpp-python, linkify-it-py, jsonschema, jinja2, contourpy, click, anyio, aiosignal, uvicorn, torch, starlette, responses, pandas, mdit-py-plugins, matplotlib, huggingface-hub, httpcore, aiohttp, transformers, httpx, fastapi, altair, accelerate, peft, gradio-client, flexgen, datasets, gradio`
when trying to launch this happens
X:\Auto-TEXT-WEBUI\gpt\text-generation-webui>python server.py --auto-devices --load-in-8bit --disk
Traceback (most recent call last):
File "X:\Auto-TEXT-WEBUI\gpt\text-generation-webui\server.py", line 17, in <module>
import gradio as gr
ModuleNotFoundError: No module named 'gradio'
X:\Auto-TEXT-WEBUI\gpt\text-generation-webui>pause
Press any key to continue . . .
@dan4in The No module named 'gradio' error is because you are not running the webui in the virtual environment. The installer failed to create that environment due to a network connection issue. Either try again some other time, or use a VPN.
@dan4in The
No module named 'gradio'error is because you are not running the webui in the virtual environment. The installer failed to create that environment due to a network connection issue. Either try again some other time, or use a VPN.
i tried over and over again it worked in the end just slow internet now it works with cuda and all Ty ♥
Still not working for me. maybe my internet connection is not good enough for it for comparison. stable diffusion only takes one try to installed flawlessly, but this "marvel" after almost 20 attempts still give me error. I'm done
I get the same CUDA Extension Not Installed error using the same vicuna_13b_4bit_128g model (I have not tried any other models). On Linux Mint (21.1), 1080Ti, 525 drivers. I succesfully ran the setup with no download errors. When I load the model, it successfully loads the model to the GPU (both by message and by checking VRAM usage with nvidia-smi)
The output of
python -m torch.utils.collect_env
clearly indicates that CUDA is available. Output (excerpted):
CUDA used to build PyTorch: 11.7 ROCM used to build PyTorch: N/A
OS: Linux Mint 21.1 (x86_64) GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0 Clang version: Could not collect CMake version: version 3.26.3 Libc version: glibc-2.35
Python version: 3.10.11 (main, Apr 20 2023, 19:02:41) [GCC 11.2.0] (64-bit runtime) Python platform: Linux-5.15.0-71-generic-x86_64-with-glibc2.35 Is CUDA available: True CUDA runtime version: 11.7.64 CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1080 Ti GPU 1: NVIDIA GeForce GTX 1050 Ti
Nvidia driver version: 525.105.17 cuDNN version: Could not collect ... [conda] pytorch-cuda 11.7 h778d358_3 pytorch ...
Is the cuDNN not found concerning?
@afoland xNul has made several changes to his installer in an attempt to fix this. Try it and see if it works: https://github.com/xNul/one-click-installers
If you don't want to fully re-install, then run python -m pip uninstall quant_cuda before running the installer and it should just re-install GPTQ-for-LLaMa.
cuDNN not being found shouldn't be an issue. It always says that on my system.
@jllllll
I retried the old installer; with the same results; then I tried the new installer you just linked above. For clarity, after downloading, I did this:
unzip one-click-installers-main.zip
cd one-click-installers-main
chmod 755 *.sh
./generate_zips.sh
cp oobabooga_linux.zip ../
cd ..
conda create -n textgen python=3.10.9
conda activate textgen
unzip oobabooga_linux.zip
cd oobabooga_linux
./start_linux.sh
Now the start script more or less immediately failed (differently from the old installer, which basically worked, just didn't pick up CUDA):
What is your GPU
A) NVIDIA B) AMD C) Apple M Series D) None (I want to run in CPU mode)
Input> A Collecting package metadata (current_repodata.json): failed
>>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
Traceback (most recent call last): File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/gateways/repodata/__init__.py", line 132, in conda_http_errors yield File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/gateways/repodata/__init__.py", line 101, in repodata response.raise_for_status() File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://conda.anaconda.org/nvidia/label/cuda-11.7.0/linux-64/current_repodata.json During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/requests/models.py", line 971, in json return complexjson.loads(self.text, **kwargs) File "/home/andrew/.local/lib/python3.10/site-packages/simplejson/__init__.py", line 514, in loads return _default_decoder.decode(s) File "/home/andrew/.local/lib/python3.10/site-packages/simplejson/decoder.py", line 386, in decode obj, end = self.raw_decode(s) File "/home/andrew/.local/lib/python3.10/site-packages/simplejson/decoder.py", line 416, in raw_decode return self.scan_once(s, idx=_w(s, idx).end()) simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/exceptions.py", line 1124, in __call__ return func(*args, **kwargs) File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/cli/main.py", line 69, in main_subshell exit_code = do_call(args, p) File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/cli/conda_argparse.py", line 91, in do_call return getattr(module, func_name)(args, parser) File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/notices/core.py", line 109, in wrapper return func(*args, **kwargs) File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/cli/main_install.py", line 20, in execute install(args, parser, 'install') File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/cli/install.py", line 264, in install unlink_link_transaction = solver.solve_for_transaction( File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/core/solve.py", line 132, in solve_for_transaction unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier, File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/core/solve.py", line 175, in solve_for_diff final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned, File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/core/solve.py", line 280, in solve_final_state ssc = self._collect_all_metadata(ssc) File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/common/io.py", line 84, in decorated return f(*args, **kwds) File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/core/solve.py", line 447, in _collect_all_metadata index, r = self._prepare(prepared_specs) File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/core/solve.py", line 1060, in _prepare reduced_index = get_reduced_index(self.prefix, self.channels, File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/core/index.py", line 267, in get_reduced_index new_records = SubdirData.query_all(spec, channels=channels, subdirs=subdirs, File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/core/subdir_data.py", line 124, in query_all result = tuple(chain.from_iterable(executor.map(subdir_query, channel_urls))) File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator yield _result_or_cancel(fs.pop()) File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel return fut.result(timeout) File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/concurrent/futures/_base.py", line 451, in result return self.__get_result() File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/core/subdir_data.py", line 113, in <lambda> subdir_query = lambda url: tuple( File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/core/subdir_data.py", line 129, in query self.load() File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/core/subdir_data.py", line 215, in load _internal_state = self._load() File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/core/subdir_data.py", line 323, in _load raw_repodata_str = self._repo.repodata(mod_etag_headers) File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/gateways/repodata/__init__.py", line 94, in repodata with conda_http_errors(self._url, filename): File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/gateways/repodata/__init__.py", line 179, in conda_http_errors raise RepodataIsEmpty( File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/conda/exceptions.py", line 461, in __init__ body = response.json() File "/home/andrew/oobabooga_linux/installer_files/conda/lib/python3.10/site-packages/requests/models.py", line 975, in json raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
$ /home/andrew/oobabooga_linux/installer_files/conda/bin/conda install -y -k pytorch[version=2,build=py3.10_cuda11.7*] torchvision torchaudio pytorch-cuda=11.7 cuda-toolkit ninja git -c pytorch -c nvidia/label/cuda-11.7.0 -c nvidiaenvironment variables: CIO_TEST=
CONDA_DEFAULT_ENV=/home/andrew/oobabooga_linux/installer_files/env CONDA_EXE=/home/andrew/oobabooga_linux/installer_files/conda/bin/conda CONDA_PREFIX=/home/andrew/oobabooga_linux/installer_files/env CONDA_PREFIX_1=/home/andrew/miniconda3 CONDA_PREFIX_2=/home/andrew/miniconda3/envs/retextgen CONDA_PROMPT_MODIFIER=(/home/andrew/oobabooga_linux/installer_files/env) CONDA_PYTHON_EXE=/home/andrew/oobabooga_linux/installer_files/conda/bin/python CONDA_ROOT=/home/andrew/oobabooga_linux/installer_files/conda CONDA_SHLVL=3 CURL_CA_BUNDLE= LD_PRELOAD= PATH=/home/andrew/oobabooga_linux/installer_files/env/bin:/home/andrew/mini conda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbi n:/bin:/usr/games:/usr/local/games:/snap/bin REQUESTS_CA_BUNDLE= SSL_CERT_FILE= XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0 XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0 active environment : /home/andrew/oobabooga_linux/installer_files/env active env location : /home/andrew/oobabooga_linux/installer_files/env shell level : 3 user config file : /home/andrew/.condarcpopulated config files : conda version : 23.1.0 conda-build version : not installed python version : 3.10.9.final.0 virtual packages : __archspec=1=x86_64 __cuda=12.0=0 __glibc=2.35=0 __linux=5.15.0=0 __unix=0=0 base environment : /home/andrew/oobabooga_linux/installer_files/conda (writable) conda av data dir : /home/andrew/oobabooga_linux/installer_files/conda/etc/conda conda av metadata url : None channel URLs : https://conda.anaconda.org/pytorch/linux-64 https://conda.anaconda.org/pytorch/noarch https://conda.anaconda.org/nvidia/label/cuda-11.7.0/linux-64 https://conda.anaconda.org/nvidia/label/cuda-11.7.0/noarch https://conda.anaconda.org/nvidia/linux-64 https://conda.anaconda.org/nvidia/noarch https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /home/andrew/oobabooga_linux/installer_files/conda/pkgs /home/andrew/.conda/pkgs envs directories : /home/andrew/oobabooga_linux/installer_files/conda/envs /home/andrew/.conda/envs platform : linux-64 user-agent : conda/23.1.0 requests/2.28.1 CPython/3.10.9 Linux/5.15.0-71-generic linuxmint/21.1 glibc/2.35 UID:GID : 1000:1000 netrc file : None offline mode : False
An unexpected error has occurred. Conda has prepared the above report.
Upload successful. Command '. "/home/andrew/oobabooga_linux/installer_files/conda/etc/profile.d/conda.sh" && conda activate "/home/andrew/oobabooga_linux/installer_files/env" && conda install -y -k pytorch[version=2,build=py3.10_cuda11.7*] torchvision torchaudio pytorch-cuda=11.7 cuda-toolkit ninja git -c pytorch -c nvidia/label/cuda-11.7.0 -c nvidia' failed with exit status code '1'. Exiting...
Done!
For what it's worth nvcc --version shows 11.5
nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2021 NVIDIA Corporation Built on Thu_Nov_18_09:45:30_PST_2021 Cuda compilation tools, release 11.5, V11.5.119 Build cuda_11.5.r11.5/compiler.30672275_0
@afoland You created a Conda environment to run an installer which creates a Conda environment? This should be all that is necessary:
unzip one-click-installers-main.zip
cd one-click-installers-main
chmod 755 *.sh
./start_linux.sh
Your environment variables indicate that you are nesting multiple Conda environments just to run this webui. This is almost certainly going to cause unpredictable behavior. The installer installs an isolated (hopefully) Miniconda installation that it uses to create and activate a Conda environment for the sole purpose of installing and running the webui.
On Windows, this issue can be somewhat mitigated by setting these environment variables in the start script:
SET "CONDA_SHLVL="
SET PYTHONNOUSERSITE=1
SET "PYTHONPATH="
I'm unsure if the Linux equivalent would achieve the same result.
BLUF: installing from conda base, following the commands, using the new one-click-installer, leads to the identical error report as my last post (leaving me with no usable installation even for non-cuda models). Installing from conda base, following the commands, using the default one-click installer (from April 28), leads to a usable installation that still fails with "CUDA not installed".
In the past I had found that the conda gymnastics were required to get it to work at all even for non-CUDA models. That said, today I installed from base and could at least get it to work on non-CUDA models.
Edit: for entertainment purposes only I deactivated conda base as well and reran both the old and new installers. Both behaved the same way: the new installer failing immediately as above, the old installer chugging along but leading finally to CUDA not installed.
So actually reading the installer's error messages, it's complaining that the url for current_repodata.json is not found:
File "/home/andrew/.local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://conda.anaconda.org/nvidia/label/cuda-12.1.0/linux-64/current_repodata.json
(I tried other versions of cuda, they all have the same issue--11.3, 1.5, 11.7)
Indeed I do find that this url does not exist when I try it in a browser. repodata.json (without the current_) does exist for all of these, however.
I don't really know what to do with this information...
So actually reading the installer's error messages, it's complaining that the url for current_repodata.json is not found:
File "/home/andrew/.local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://conda.anaconda.org/nvidia/label/cuda-12.1.0/linux-64/current_repodata.json
(I tried other versions of cuda, they all have the same issue--11.3, 1.5, 11.7)
Indeed I do find that this url does not exist when I try it in a browser. repodata.json (without the current_) does exist for all of these, however.
I don't really know what to do with this information...
I've been searching around for this issue for a while and can only assume that this is an old, unsolved bug in Conda specific to linux-64.
Try using this command in webui.py:
conda install -y -k --repodata-fn repodata.json --repodata-fn current_repodata.json pytorch[version=2,build=py3.10_cuda11.7*] torchvision torchaudio pytorch-cuda=11.7 cuda-toolkit ninja git -c pytorch -c nvidia/label/cuda-11.7.0 -c nvidia
I am in exactly the same position (Arch x64) and have tried everything to get the WebUI to work with CUDA.
I tried this and am still getting errors myself: File "/home/david/text-generation-webui/repositories/GPTQ-for-LLaMa/quant.py", line 426, in forward quant_cuda.vecquant4matmul(x, self.qweight, y, self.scales, self.qzeros, self.groupsize) NameError: name 'quant_cuda' is not defined
Resolved! (for me)
Unfortunately not very insightfully.
I went to an absolutely fresh install of linux mint 21.1. Then I:
-
Installed most recent nvidia driver (530 as of today.) Rebooted.
-
Manually installed cuda-11.7 (from NVIDIA website, only the debian-network option worked) immediately. Rebooted.
-
Added the cuda-11.7 to path and ld_library path in /bashrc and sourced .bashrc . (Note for linux-mint users, there appears to be a bug in linux mint which may prevent ld_library in bashrc being executed at start-up.)
-
Ran the start_linux script (note that this script did appear--based on output along the way--to recognize that the current_repodata wasn't present and backed off to repodata)
Uninformed speculation: It is my guess that when cuda 11.7 becomes the default in the repos for cuda (it's 11.5 today) a lot of this trouble will go away.
Thanks to all who took the time to look into it!
I'm getting this when loading a model on WSL Debian. I've installed CUDA, and it's a fresh webui installation. It works, but it's much slower than Windows.
2023-06-18 00:05:11 WARNING:CUDA extension not installed.
2023-06-18 00:05:12 WARNING:The safetensors archive passed at models/TheBloke_WizardCoder-15B-1.0-GPTQ/gptq_model-4bit-128g.safetensors does not contain metadata. Make sure to save your model with the `save_pretrained` method. Defaulting to 'pt' metadata.
2023-06-18 00:05:25 WARNING:GPTBigCodeGPTQForCausalLM hasn't fused attention module yet, will skip inject fused attention.
2023-06-18 00:05:25 WARNING:GPTBigCodeGPTQForCausalLM hasn't fused mlp module yet, will skip inject fused mlp.
@missionfloyd - 100% same case here. Same model /7b though/ - GPU memory is in use, however when I do any query over langchain, my CPU usage hits 100%.
This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}