text-generation-webui
text-generation-webui copied to clipboard
Published
20 hours ago •
oobabooga
oobabooga
Model Crashes upon interaction with KeyError:'model.layers... , Cuda kernel compilation failed during single click installer
Describe the bug
I can load the Ui successfully, Model loads successfully, but any attempt to generate things fails with a key error.
Is there an existing issue for this?
- [X] I have searched the existing issues
Reproduction
Run the single click installer on windows, observe
C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\torch\utils\cpp_extension.py:359: UserWarning: Error checking compiler version for cl: [WinError 2] Das System kann die angegebene Datei nicht finden warnings.warn(f'Error checking compiler version for {compiler}: {error}') error: [WinError 2] Das System kann die angegebene Datei nicht finden CUDA kernel compilation failed. Attempting installation with wheel. Collecting quant-cuda==0.0.0
(ignore the folder name, this was not installed or launched with seait, just lives in the same folder)
Load the model (model used: gpt-x-alpaca-13b-native-4bit-128g-cuda.pt), but upon any interaction it will fail with either a memory error, or a KeyError with flag --gpu-memory 6.
Debugging steps taken
Tried manual install on windows with the reddit post. Same compilation error. Tried upgrading the seait installed version (which uses venv instead of conda). Installed cuda, first 12.1.1, then downgrade to 11.7. Installed Visual Studio Build Tools C++, first 2022, then downgrade to 2019. Made sure C++ is actually installed. cl.exe works within the environment, no PATH variable is set, as is default. Installed cuDNN No success anywhere. Resolved other errors but always got stuck on {compiler}: {error}
possibly related issue could be #1273 , though I am seeing a different error.
Screenshot
No response
Logs
--- INSTALL EXCERPT ---
Successfully uninstalled datasets-2.11.0
Successfully installed datasets-2.10.1 transformers-4.28.0
running install
C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\setuptools\command\easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running bdist_egg
running egg_info
creating quant_cuda.egg-info
writing quant_cuda.egg-info\PKG-INFO
writing dependency_links to quant_cuda.egg-info\dependency_links.txt
writing top-level names to quant_cuda.egg-info\top_level.txt
writing manifest file 'quant_cuda.egg-info\SOURCES.txt'
reading manifest file 'quant_cuda.egg-info\SOURCES.txt'
writing manifest file 'quant_cuda.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
running build_ext
C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\torch\utils\cpp_extension.py:359: UserWarning: Error checking compiler version for cl: [WinError 2] Das System kann die angegebene Datei nicht finden
warnings.warn(f'Error checking compiler version for {compiler}: {error}')
error: [WinError 2] Das System kann die angegebene Datei nicht finden
CUDA kernel compilation failed.
Attempting installation with wheel.
Collecting quant-cuda==0.0.0
Using cached https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl (398 kB)
Installing collected packages: quant-cuda
Successfully installed quant-cuda-0.0.0
Select the model that you want to download:
A) OPT 6.7B
B) OPT 2.7B
--- FULL LOG OF MODEL LOAD AND CRASH ON INTERACTION ---
Gradio HTTP request redirected to localhost :)
bin C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\bitsandbytes\libbitsandbytes_cuda117.dll
Loading gpt4-x-alpaca-13b-native-4bit-128g...
Found the following quantized model: models\gpt4-x-alpaca-13b-native-4bit-128g\gpt-x-alpaca-13b-native-4bit-128g-cuda.pt
Loading model ...
Done.
Using the following device map for the quantized model: {'model.embed_tokens': 0, 'model.layers.0': 0, 'model.layers.1': 0, 'model.layers.2': 0, 'model.layers.3': 0, 'model.layers.4': 0, 'model.layers.5': 0, 'model.layers.6': 0, 'model.layers.7': 0, 'model.layers.8': 0, 'model.layers.9': 0, 'model.layers.10': 0, 'model.layers.11': 0, 'model.layers.12': 0, 'model.layers.13': 0, 'model.layers.14': 0, 'model.layers.15': 0, 'model.layers.16': 0, 'model.layers.17': 0, 'model.layers.18': 0, 'model.layers.19': 0, 'model.layers.20': 0, 'model.layers.21': 0, 'model.layers.22': 0, 'model.layers.23': 0, 'model.layers.24': 0, 'model.layers.25': 0, 'model.layers.26': 0, 'model.layers.27': 0, 'model.layers.28': 0, 'model.layers.29': 0, 'model.layers.30': 0, 'model.layers.31': 0, 'model.layers.32': 0, 'model.layers.33': 'cpu', 'model.layers.34': 'cpu', 'model.layers.35': 'cpu', 'model.layers.36': 'cpu', 'model.layers.37': 'cpu', 'model.layers.38': 'cpu', 'model.layers.39': 'cpu', 'model.norm': 'cpu', 'lm_head': 'cpu'}
Loaded the model in 37.73 seconds.
Loading the extension "gallery"... Ok.
Running on local URL: http://127.0.0.1:7860
To create a public link, set share=True in launch().
Traceback (most recent call last):
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\text-generation-webui\modules\callbacks.py", line 66, in gentask
ret = self.mfunc(callback=_callback, **self.kwargs)
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\text-generation-webui\modules\text_generation.py", line 290, in generate_with_callback
shared.model.generate(**kwargs)
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\transformers\generation\utils.py", line 1485, in generate
return self.sample(
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\transformers\generation\utils.py", line 2524, in sample
outputs = self(
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\transformers\models\llama\modeling_llama.py", line 687, in forward
outputs = self.model(
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\transformers\models\llama\modeling_llama.py", line 577, in forward
layer_outputs = decoder_layer(
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\transformers\models\llama\modeling_llama.py", line 292, in forward
System Info
Windows 10 Pro
Ryzen 5 3600
16 GB RAM, 3200 MHZ DDR4
Nvidia GeForce RTX 2060 Super (8 GB)
Logs created with start_windows.bat and model launch parameters: --chat --model-menu --auto-devices --wbits 4 --groupsize 128 --gpu-memory 6
Can you verify that a folder starting with "quant_cuda" exists in your installer_files/env/libs/site-packages folder? If it doesn't, some fixes I've made to the installer might help https://github.com/xNul/one-click-installers
@xNul I have had a chance to have a look now
Under installer_files/env/libs/ there is no folder called site-packages, in fact there is no folder within libs at all.
However, under env/Lib there is a site-packages folder, within which I could find quant_cuda-0.0.0.dist-info but no quant-cuda.
I will attempt to use your fixed installer.
I have tested your installer. It gave me a different looking error, but it still failed to compile quant-cuda
Successfully installed datasets-2.10.1 transformers-4.28.0
Processing c:\seait_supereasyaiinstallertool\text-generation-webui\text-generation-webui\repositories\gptq-for-llama
Preparing metadata (setup.py) ... done
Building wheels for collected packages: quant-cuda
Building wheel for quant-cuda (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [6 lines of output]
running bdist_wheel
running build
running build_ext
C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\torch\utils\cpp_extension.py:359: UserWarning: Error checking compiler version for cl: [WinError 2] Das System kann die angegebene Datei nicht finden
warnings.warn(f'Error checking compiler version for {compiler}: {error}')
error: [WinError 2] Das System kann die angegebene Datei nicht finden
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for quant-cuda
Running setup.py clean for quant-cuda
Failed to build quant-cuda
Installing collected packages: quant-cuda
Running setup.py install for quant-cuda ... error
error: subprocess-exited-with-error
× Running setup.py install for quant-cuda did not run successfully.
│ exit code: 1
╰─> [8 lines of output]
running install
C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_ext
C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\torch\utils\cpp_extension.py:359: UserWarning: Error checking compiler version for cl: [WinError 2] Das System kann die angegebene Datei nicht finden
warnings.warn(f'Error checking compiler version for {compiler}: {error}')
error: [WinError 2] Das System kann die angegebene Datei nicht finden
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure
× Encountered error while trying to install package.
╰─> quant-cuda
note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
ERROR: GPTQ CUDA kernel compilation failed.
Attempting installation with wheel.
Collecting quant-cuda==0.0.0
Using cached https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl (398 kB)
Installing collected packages: quant-cuda
Successfully installed quant-cuda-0.0.0
Wheel installation success!
Continuing with install..
Select the model that you want to download:
A) OPT 6.7B
B) OPT 2.7B
looking in env/Lib again, there is no more quant-cuda-0.0.0.dist-info and also no quant-cuda folder. Overall, there appear to be less folders total in Lib than there used to be.
I used your installer by downloading the repository as a zip, grabbing the contents into my folder, and running start_windows.bat
@HB-Stratos you're right, I typo'd there. quant_cuda should be under installer_files/env/Lib/site-packages.
This is very strange. Under the new install using my one-click installer, could you double check that there isn't a folder starting with quant_cuda under installer_files/env/Lib/site-packages? Make sure it has the underscore and not the apostrophe.
It's ok for the compilation to fail as long as that folder exists at that location. If the folder does exist, then you can verify everything is working by downloading the model https://huggingface.co/wcde/llama-7b-4bit-gr128 and running it
I just had a look, I checked the wrong folder earlier. Now I checked text-generation-webui\installer_files\env\Lib\site-packages/ , and quant_cuda dist info exists again, but there is no quant_cuda folder.
Here's a dir of that folder, sorry for the german.
Volume in Laufwerk C: hat keine Bezeichnung.
Volumeseriennummer: 8AF2-8721
Verzeichnis von C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\Lib\site-packages
01.05.2023 14:52 <DIR> .
01.05.2023 14:52 <DIR> ..
01.05.2023 14:49 <DIR> accelerate
01.05.2023 14:49 <DIR> accelerate-0.18.0.dist-info
01.05.2023 14:49 <DIR> aiofiles
01.05.2023 14:49 <DIR> aiofiles-23.1.0.dist-info
01.05.2023 14:49 <DIR> aiohttp
01.05.2023 14:49 <DIR> aiohttp-3.8.4.dist-info
01.05.2023 14:49 <DIR> aiosignal
01.05.2023 14:49 <DIR> aiosignal-1.3.1.dist-info
01.05.2023 14:49 <DIR> altair
01.05.2023 14:49 <DIR> altair-4.2.2.dist-info
01.05.2023 14:50 <DIR> antlr4
01.05.2023 14:50 <DIR> antlr4_python3_runtime-4.9.3.dist-info
01.05.2023 14:49 <DIR> anyio
01.05.2023 14:49 <DIR> anyio-3.6.2.dist-info
01.05.2023 14:50 <DIR> asttokens
01.05.2023 14:50 <DIR> asttokens-2.2.1.dist-info
01.05.2023 14:49 <DIR> async_timeout
01.05.2023 14:49 <DIR> async_timeout-4.0.2.dist-info
01.05.2023 14:49 <DIR> attr
01.05.2023 14:49 <DIR> attrs
01.05.2023 14:49 <DIR> attrs-23.1.0.dist-info
01.05.2023 14:50 <DIR> backcall
01.05.2023 14:50 <DIR> backcall-0.2.0.dist-info
01.05.2023 14:50 <DIR> beautifulsoup4-4.12.2.dist-info
01.05.2023 14:48 <DIR> bitsandbytes
01.05.2023 14:48 <DIR> bitsandbytes-0.38.1.dist-info
01.05.2023 14:50 <DIR> blinker
01.05.2023 14:50 <DIR> blinker-1.6.2.dist-info
01.05.2023 14:48 <DIR> brotli
01.05.2023 14:48 <DIR> brotlipy-0.7.0-py3.10.egg-info
01.05.2023 14:50 <DIR> bs4
01.05.2023 14:48 <DIR> certifi
01.05.2023 14:48 <DIR> certifi-2022.12.7.dist-info
01.05.2023 14:48 <DIR> cffi
01.05.2023 14:48 <DIR> cffi-1.15.1.dist-info
01.05.2023 14:48 <DIR> charset_normalizer
01.05.2023 14:48 <DIR> charset_normalizer-2.0.4.dist-info
01.05.2023 14:49 <DIR> click
01.05.2023 14:49 <DIR> click-8.1.3.dist-info
01.05.2023 14:49 <DIR> colorama
01.05.2023 14:49 <DIR> colorama-0.4.6.dist-info
01.05.2023 14:49 <DIR> contourpy
01.05.2023 14:49 <DIR> contourpy-1.0.7.dist-info
01.05.2023 14:48 <DIR> cryptography
01.05.2023 14:48 <DIR> cryptography-39.0.1.dist-info
01.05.2023 14:49 <DIR> cycler-0.11.0.dist-info
01.05.2023 14:49 14.519 cycler.py
01.05.2023 14:51 <DIR> datasets
01.05.2023 14:51 <DIR> datasets-2.10.1.dist-info
01.05.2023 14:49 <DIR> dateutil
01.05.2023 14:50 <DIR> decorator-5.1.1.dist-info
01.05.2023 14:50 16.752 decorator.py
01.05.2023 14:50 <DIR> deep_translator
01.05.2023 14:50 <DIR> deep_translator-1.9.2.dist-info
01.05.2023 14:49 <DIR> dill
01.05.2023 14:49 <DIR> dill-0.3.6.dist-info
17.04.2023 12:08 152 distutils-precedence.pth
01.05.2023 14:50 <DIR> docopt-0.6.2.dist-info
01.05.2023 14:50 19.946 docopt.py
01.05.2023 14:50 <DIR> elevenlabslib
01.05.2023 14:50 <DIR> elevenlabslib-0.6.0.dist-info
01.05.2023 14:49 <DIR> entrypoints-0.4.dist-info
01.05.2023 14:49 8.264 entrypoints.py
01.05.2023 14:50 <DIR> executing
01.05.2023 14:50 <DIR> executing-1.2.0.dist-info
01.05.2023 14:50 <DIR> fastapi
01.05.2023 14:50 <DIR> fastapi-0.95.1.dist-info
01.05.2023 14:51 <DIR> ffmpeg
01.05.2023 14:51 <DIR> ffmpeg-1.4.dist-info
01.05.2023 14:51 <DIR> ffmpeg_python-0.2.0.dist-info
01.05.2023 14:49 <DIR> ffmpy-0.3.0.dist-info
01.05.2023 14:49 8.461 ffmpy.py
01.05.2023 14:48 <DIR> filelock
01.05.2023 14:48 <DIR> filelock-3.9.0.dist-info
01.05.2023 14:50 <DIR> flask
01.05.2023 14:50 <DIR> Flask-2.3.1.dist-info
01.05.2023 14:50 <DIR> flask_cloudflared-0.0.12.dist-info
01.05.2023 14:50 7.325 flask_cloudflared.py
01.05.2023 14:50 <DIR> flexgen
01.05.2023 14:50 <DIR> flexgen-0.1.7.dist-info
01.05.2023 14:49 <DIR> fontTools
01.05.2023 14:49 <DIR> fonttools-4.39.3.dist-info
01.05.2023 14:49 <DIR> frozenlist
01.05.2023 14:49 <DIR> frozenlist-1.3.3.dist-info
01.05.2023 14:49 <DIR> fsspec
01.05.2023 14:49 <DIR> fsspec-2023.4.0.dist-info
01.05.2023 14:48 <DIR> functorch
01.05.2023 14:51 <DIR> future
01.05.2023 14:51 <DIR> future-0.18.3.dist-info
01.05.2023 14:50 <DIR> gradio
01.05.2023 14:50 <DIR> gradio-3.25.0.dist-info
01.05.2023 14:50 <DIR> gradio_client
01.05.2023 14:50 <DIR> gradio_client-0.1.4.dist-info
01.05.2023 14:49 <DIR> h11
01.05.2023 14:49 <DIR> h11-0.14.0.dist-info
01.05.2023 14:49 <DIR> httpcore
01.05.2023 14:49 <DIR> httpcore-0.17.0.dist-info
01.05.2023 14:50 <DIR> httpx
01.05.2023 14:50 <DIR> httpx-0.24.0.dist-info
01.05.2023 14:49 <DIR> huggingface_hub
01.05.2023 14:49 <DIR> huggingface_hub-0.14.1.dist-info
01.05.2023 14:48 <DIR> idna
01.05.2023 14:48 <DIR> idna-3.4.dist-info
01.05.2023 14:50 <DIR> IPython
01.05.2023 14:50 <DIR> ipython-8.13.1.dist-info
12.11.2022 01:22 11.199 isympy.py
01.05.2023 14:50 <DIR> itsdangerous
01.05.2023 14:50 <DIR> itsdangerous-2.1.2.dist-info
01.05.2023 14:50 <DIR> jedi
01.05.2023 14:50 <DIR> jedi-0.18.2.dist-info
01.05.2023 14:48 <DIR> jinja2
01.05.2023 14:48 <DIR> Jinja2-3.1.2.dist-info
01.05.2023 14:49 <DIR> jsonschema
01.05.2023 14:49 <DIR> jsonschema-4.17.3.dist-info
01.05.2023 14:49 <DIR> kiwisolver
01.05.2023 14:49 <DIR> kiwisolver-1.4.4.dist-info
01.05.2023 14:51 <DIR> libfuturize
01.05.2023 14:51 <DIR> libpasteurize
01.05.2023 14:49 <DIR> linkify_it
01.05.2023 14:49 <DIR> linkify_it_py-2.0.0.dist-info
01.05.2023 14:49 <DIR> llama_cpp
01.05.2023 14:49 <DIR> llama_cpp_python-0.1.36.dist-info
01.05.2023 14:51 <DIR> llvmlite
01.05.2023 14:51 <DIR> llvmlite-0.39.1.dist-info
01.05.2023 14:49 <DIR> markdown
01.05.2023 14:49 <DIR> Markdown-3.4.3.dist-info
01.05.2023 14:49 <DIR> markdown_it
01.05.2023 14:49 <DIR> markdown_it_py-2.2.0.dist-info
01.05.2023 14:48 <DIR> markupsafe
01.05.2023 14:48 <DIR> MarkupSafe-2.1.1.dist-info
01.05.2023 14:49 <DIR> matplotlib
01.05.2023 14:49 570 matplotlib-3.7.1-py3.10-nspkg.pth
01.05.2023 14:49 <DIR> matplotlib-3.7.1.dist-info
01.05.2023 14:49 <DIR> matplotlib.libs
01.05.2023 14:50 <DIR> matplotlib_inline
01.05.2023 14:50 <DIR> matplotlib_inline-0.1.6.dist-info
01.05.2023 14:49 <DIR> mdit_py_plugins
01.05.2023 14:49 <DIR> mdit_py_plugins-0.3.3.dist-info
01.05.2023 14:49 <DIR> mdurl
01.05.2023 14:49 <DIR> mdurl-0.1.2.dist-info
01.05.2023 14:48 <DIR> mkl
01.05.2023 14:48 <DIR> mkl_fft
01.05.2023 14:48 <DIR> mkl_fft-1.3.1-py3.10.egg-info
01.05.2023 14:48 <DIR> mkl_random
01.05.2023 14:48 <DIR> mkl_random-1.2.2.dist-info
01.05.2023 14:48 <DIR> mkl_service-2.4.0-py3.10.egg-info
01.05.2023 14:51 <DIR> more_itertools
01.05.2023 14:51 <DIR> more_itertools-9.1.0.dist-info
01.05.2023 14:49 <DIR> mpl_toolkits
01.05.2023 14:48 <DIR> mpmath
01.05.2023 14:48 <DIR> mpmath-1.2.1-py3.10.egg-info
01.05.2023 14:49 <DIR> multidict
01.05.2023 14:49 <DIR> multidict-6.0.4.dist-info
01.05.2023 14:49 <DIR> multipart
01.05.2023 14:49 <DIR> multiprocess
01.05.2023 14:49 <DIR> multiprocess-0.70.14.dist-info
01.05.2023 14:48 <DIR> networkx
01.05.2023 14:48 <DIR> networkx-2.8.4.dist-info
01.05.2023 14:50 <DIR> num2words
01.05.2023 14:50 <DIR> num2words-0.5.12.dist-info
01.05.2023 14:51 <DIR> numba
01.05.2023 14:51 <DIR> numba-0.56.4.dist-info
01.05.2023 14:51 <DIR> numpy
01.05.2023 14:51 <DIR> numpy-1.23.5.dist-info
01.05.2023 14:48 <DIR> nvfuser
01.05.2023 14:50 <DIR> omegaconf
01.05.2023 14:50 <DIR> omegaconf-2.3.0.dist-info
01.05.2023 14:51 <DIR> openai_whisper-20230314.dist-info
01.05.2023 14:48 <DIR> OpenSSL
01.05.2023 14:49 <DIR> orjson
01.05.2023 14:49 <DIR> orjson-3.8.11.dist-info
01.05.2023 14:49 <DIR> packaging
01.05.2023 14:49 <DIR> packaging-23.1.dist-info
01.05.2023 14:49 <DIR> pandas
01.05.2023 14:49 <DIR> pandas-2.0.1.dist-info
01.05.2023 14:50 <DIR> parso
01.05.2023 14:50 <DIR> parso-0.8.3.dist-info
01.05.2023 14:51 <DIR> past
01.05.2023 14:50 <DIR> peft
01.05.2023 14:50 <DIR> peft-0.3.0.dev0.dist-info
01.05.2023 14:50 <DIR> pickleshare-0.7.5.dist-info
01.05.2023 14:50 9.942 pickleshare.py
01.05.2023 14:49 <DIR> PIL
01.05.2023 14:49 <DIR> Pillow-9.5.0.dist-info
01.05.2023 14:41 <DIR> pip
01.05.2023 14:41 <DIR> pip-23.0.1-py3.10.egg-info
01.05.2023 14:41 <DIR> pkg_resources
01.05.2023 14:50 <DIR> prompt_toolkit
01.05.2023 14:50 <DIR> prompt_toolkit-3.0.38.dist-info
01.05.2023 14:49 <DIR> psutil
01.05.2023 14:49 <DIR> psutil-5.9.5.dist-info
01.05.2023 14:49 <DIR> pulp
01.05.2023 14:49 <DIR> PuLP-2.7.0.dist-info
01.05.2023 14:50 <DIR> pure_eval
01.05.2023 14:50 <DIR> pure_eval-0.2.2.dist-info
01.05.2023 14:49 30.720 pvectorc.cp310-win_amd64.pyd
01.05.2023 14:49 <DIR> pyarrow
01.05.2023 14:49 <DIR> pyarrow-11.0.0.dist-info
01.05.2023 14:48 <DIR> pycparser
01.05.2023 14:48 <DIR> pycparser-2.21.dist-info
01.05.2023 14:49 <DIR> pydantic
01.05.2023 14:49 <DIR> pydantic-1.10.7.dist-info
01.05.2023 14:50 <DIR> pydevd_plugins
01.05.2023 14:49 <DIR> pydub
01.05.2023 14:49 <DIR> pydub-0.25.1.dist-info
01.05.2023 14:50 <DIR> pygments
01.05.2023 14:50 <DIR> Pygments-2.15.1.dist-info
01.05.2023 14:49 93 pylab.py
01.05.2023 14:48 <DIR> pyOpenSSL-23.0.0.dist-info
01.05.2023 14:49 <DIR> pyparsing
01.05.2023 14:49 <DIR> pyparsing-3.0.9.dist-info
01.05.2023 14:49 <DIR> pyrsistent
01.05.2023 14:49 <DIR> pyrsistent-0.19.3.dist-info
01.05.2023 14:48 <DIR> PySocks-1.7.1.dist-info
01.05.2023 14:49 <DIR> python_dateutil-2.8.2.dist-info
01.05.2023 14:49 <DIR> python_multipart-0.0.6.dist-info
01.05.2023 14:49 <DIR> pytz
01.05.2023 14:49 <DIR> pytz-2023.3.dist-info
01.05.2023 14:49 <DIR> PyYAML-6.0.dist-info
01.05.2023 14:52 <DIR> quant_cuda-0.0.0.dist-info
01.05.2023 14:52 1.399.296 quant_cuda.cp310-win_amd64.pyd
04.04.2023 23:57 119 README.txt
01.05.2023 14:49 <DIR> regex
01.05.2023 14:49 <DIR> regex-2023.3.23.dist-info
01.05.2023 14:48 <DIR> requests
01.05.2023 14:48 <DIR> requests-2.29.0.dist-info
01.05.2023 14:49 <DIR> responses
01.05.2023 14:49 <DIR> responses-0.18.0.dist-info
01.05.2023 14:49 <DIR> rwkv
01.05.2023 14:49 <DIR> rwkv-0.7.3.dist-info
01.05.2023 14:49 <DIR> safetensors
01.05.2023 14:49 <DIR> safetensors-0.3.0.dist-info
01.05.2023 14:49 <DIR> semantic_version
01.05.2023 14:49 <DIR> semantic_version-2.10.0.dist-info
01.05.2023 14:49 <DIR> sentencepiece
01.05.2023 14:49 <DIR> sentencepiece-0.1.98.dist-info
01.05.2023 14:41 <DIR> setuptools
01.05.2023 14:41 <DIR> setuptools-66.0.0-py3.10.egg-info
01.05.2023 14:48 <DIR> six-1.16.0.dist-info
14.02.2022 23:59 34.549 six.py
01.05.2023 14:49 <DIR> sniffio
01.05.2023 14:49 <DIR> sniffio-1.3.0.dist-info
13.01.2022 17:56 31.086 socks.py
13.01.2022 17:56 3.966 sockshandler.py
01.05.2023 14:50 <DIR> sounddevice-0.4.6.dist-info
01.05.2023 14:50 107.702 sounddevice.py
01.05.2023 14:50 <DIR> soundfile-0.12.1.dist-info
01.05.2023 14:50 60.756 soundfile.py
01.05.2023 14:50 <DIR> soupsieve
01.05.2023 14:50 <DIR> soupsieve-2.4.1.dist-info
01.05.2023 14:51 <DIR> SpeechRecognition-3.9.0.dist-info
01.05.2023 14:51 <DIR> speech_recognition
01.05.2023 14:50 <DIR> stack_data
01.05.2023 14:50 <DIR> stack_data-0.6.2.dist-info
01.05.2023 14:49 <DIR> starlette
01.05.2023 14:49 <DIR> starlette-0.26.1.dist-info
01.05.2023 14:48 <DIR> sympy
01.05.2023 14:48 <DIR> sympy-1.11.1.dist-info
01.05.2023 14:51 <DIR> tiktoken
01.05.2023 14:51 <DIR> tiktoken-0.3.1.dist-info
01.05.2023 14:51 <DIR> tiktoken_ext
01.05.2023 14:49 <DIR> tlz
01.05.2023 14:49 <DIR> tokenizers
01.05.2023 14:49 <DIR> tokenizers-0.13.3.dist-info
01.05.2023 14:49 <DIR> toolz
01.05.2023 14:49 <DIR> toolz-0.12.0.dist-info
01.05.2023 14:48 <DIR> torch
01.05.2023 14:48 <DIR> torch-2.0.0-py3.10.egg-info
01.05.2023 14:48 <DIR> torchaudio
01.05.2023 14:48 <DIR> torchaudio-2.0.0-py3.10.egg-info
01.05.2023 14:48 <DIR> torchgen
01.05.2023 14:48 <DIR> torchvision
01.05.2023 14:48 <DIR> torchvision-0.15.0-py3.10.egg-info
01.05.2023 14:49 <DIR> tqdm
01.05.2023 14:49 <DIR> tqdm-4.65.0.dist-info
01.05.2023 14:50 <DIR> traitlets
01.05.2023 14:50 <DIR> traitlets-5.9.0.dist-info
01.05.2023 14:51 <DIR> transformers
01.05.2023 14:51 <DIR> transformers-4.28.0.dist-info
01.05.2023 14:50 <DIR> typing-3.7.4.3.dist-info
01.05.2023 14:50 81.922 typing.py
01.05.2023 14:48 <DIR> typing_extensions-4.5.0.dist-info
19.04.2023 23:27 84.065 typing_extensions.py
01.05.2023 14:49 <DIR> tzdata
01.05.2023 14:49 <DIR> tzdata-2023.3.dist-info
01.05.2023 14:49 <DIR> uc_micro
01.05.2023 14:49 <DIR> uc_micro_py-1.0.1.dist-info
01.05.2023 14:48 <DIR> urllib3
01.05.2023 14:48 <DIR> urllib3-1.26.15.dist-info
01.05.2023 14:49 <DIR> uvicorn
01.05.2023 14:49 <DIR> uvicorn-0.22.0.dist-info
01.05.2023 14:50 <DIR> wcwidth
01.05.2023 14:50 <DIR> wcwidth-0.2.6.dist-info
01.05.2023 14:49 <DIR> websockets
01.05.2023 14:49 <DIR> websockets-11.0.2.dist-info
01.05.2023 14:50 <DIR> werkzeug
01.05.2023 14:50 <DIR> Werkzeug-2.3.2.dist-info
01.05.2023 14:41 <DIR> wheel
01.05.2023 14:41 <DIR> wheel-0.38.4-py3.10.egg-info
01.05.2023 14:51 <DIR> whisper
01.05.2023 14:48 <DIR> win_inet_pton-1.1.0.dist-info
20.01.2022 08:01 4.035 win_inet_pton.py
01.05.2023 14:49 <DIR> xxhash
01.05.2023 14:49 <DIR> xxhash-3.2.0.dist-info
01.05.2023 14:49 <DIR> yaml
01.05.2023 14:49 <DIR> yarl
01.05.2023 14:49 <DIR> yarl-1.9.2.dist-info
07.12.2022 16:32 163.840 _cffi_backend.cp310-win_amd64.pyd
01.05.2023 14:41 <DIR> _distutils_hack
01.05.2023 14:49 <DIR> _multiprocess
01.05.2023 14:49 24 _pyrsistent_version.py
01.05.2023 14:50 13.636 _sounddevice.py
01.05.2023 14:50 <DIR> _sounddevice_data
01.05.2023 14:50 5.561 _soundfile.py
01.05.2023 14:50 <DIR> _soundfile_data
01.05.2023 14:49 <DIR> _yaml
01.05.2023 14:50 <DIR> __pycache__
26 Datei(en), 2.118.500 Bytes
293 Verzeichnis(se), 426.256.703.488 Bytes frei```
@HB-Stratos this is exactly what we're looking for. Cool, now let's see if it runs a model. Get to the download model menu via the install or you can open cmd_windows.bat, use the command cd text-generation-webui, and then use the command python download-model.py. Once there, use the L option and enter this model "wcde/llama-7b-4bit-gr128". Once it finishes downloading, you can try running the model with webui. If it works, you're all set!
It's unfortunate the compilation method didn't work though. There's probably something wrong with the installer.
The llama 7b model works. My model still fails with a key error, or an out of ram if the flags aren't set right. Used the same flags as in the initial post this time, seems to have the same issues. What could cause this?
log:
Gradio HTTP request redirected to localhost :)
bin C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\bitsandbytes\libbitsandbytes_cuda117.dll
The following models are available:
1. gpt4-x-alpaca-13b-native-4bit-128g
2. wcde_llama-7b-4bit-gr128
Which one do you want to load? 1-2
1
Loading gpt4-x-alpaca-13b-native-4bit-128g...
Found the following quantized model: models\gpt4-x-alpaca-13b-native-4bit-128g\gpt-x-alpaca-13b-native-4bit-128g-cuda.pt
Loading model ...
Done.
Using the following device map for the quantized model: {'model.embed_tokens': 0, 'model.layers.0': 0, 'model.layers.1': 0, 'model.layers.2': 0, 'model.layers.3': 0, 'model.layers.4': 0, 'model.layers.5': 0, 'model.layers.6': 0, 'model.layers.7': 0, 'model.layers.8': 0, 'model.layers.9': 0, 'model.layers.10': 0, 'model.layers.11': 0, 'model.layers.12': 0, 'model.layers.13': 0, 'model.layers.14': 0, 'model.layers.15': 0, 'model.layers.16': 0, 'model.layers.17': 0, 'model.layers.18': 0, 'model.layers.19': 0, 'model.layers.20': 0, 'model.layers.21': 0, 'model.layers.22': 0, 'model.layers.23': 0, 'model.layers.24': 0, 'model.layers.25': 0, 'model.layers.26': 0, 'model.layers.27': 0, 'model.layers.28': 0, 'model.layers.29': 0, 'model.layers.30': 0, 'model.layers.31': 0, 'model.layers.32': 0, 'model.layers.33': 'cpu', 'model.layers.34': 'cpu', 'model.layers.35': 'cpu', 'model.layers.36': 'cpu', 'model.layers.37': 'cpu', 'model.layers.38': 'cpu', 'model.layers.39': 'cpu', 'model.norm': 'cpu', 'lm_head': 'cpu'}
Loaded the model in 44.35 seconds.
Loading the extension "gallery"... Ok.
Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.
Traceback (most recent call last):
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\text-generation-webui\modules\callbacks.py", line 66, in gentask
ret = self.mfunc(callback=_callback, **self.kwargs)
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\text-generation-webui\modules\text_generation.py", line 290, in generate_with_callback
shared.model.generate(**kwargs)
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\transformers\generation\utils.py", line 1485, in generate
return self.sample(
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\transformers\generation\utils.py", line 2524, in sample
outputs = self(
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\transformers\models\llama\modeling_llama.py", line 687, in forward
outputs = self.model(
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\transformers\models\llama\modeling_llama.py", line 577, in forward
layer_outputs = decoder_layer(
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\transformers\models\llama\modeling_llama.py", line 292, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
output = old_forward(*args, **kwargs)
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\transformers\models\llama\modeling_llama.py", line 196, in forward
query_states = self.q_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\accelerate\hooks.py", line 160, in new_forward
args, kwargs = module._hf_hook.pre_forward(module, *args, **kwargs)
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\accelerate\hooks.py", line 280, in pre_forward
set_module_tensor_to_device(module, name, self.execution_device, value=self.weights_map[name])
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\accelerate\utils\offload.py", line 123, in __getitem__
return self.dataset[f"{self.prefix}{key}"]
File "C:\seait_SuperEasyAIInstallerTool\text-generation-webui\installer_files\env\lib\site-packages\accelerate\utils\offload.py", line 170, in __getitem__
weight_info = self.index[key]
KeyError: 'model.layers.33.self_attn.q_proj.wf1'
Output generated in 2.76 seconds (0.00 tokens/s, 0 tokens, context 37, seed 2043276947)
... if I am reading this right it is loading the model on the cpu, but taskmanager is telling me it is loading into vram. What is going on here?
@HB-Stratos I've seen a lot of problems with this model. It looks like there's an incompatibility between how the model was built and how it's being used by the transformers library. My guess is that the model was made for a different version of transformers
This issue has been closed due to inactivity for 6 weeks. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}