text-generation-webui
text-generation-webui copied to clipboard
oobabooga
AssertionError: Torch not compiled with CUDA enabled
Describe the bug
After completing a fresh install using the windows one click installer, I am receiving the following error whenever I run the start_windows script:
AssertionError: Torch not compiled with CUDA enabled
Is there an existing issue for this?
- [X] I have searched the existing issues
Reproduction
- Download current version of Windows one-click installer
- Install for NVIDIA system.
- Select any model.
- Attempt to run start_windows script
Screenshot
No response
Logs
`Traceback (most recent call last):
File "C:\Users\<user>\Downloads\oobabooga_windows\oobabooga_windows\text-generation-webui\server.py", line 917, in <module>
shared.model, shared.tokenizer = load_model(shared.model_name)
File "C:\Users\<user>\Downloads\oobabooga_windows\oobabooga_windows\text-generation-webui\modules\models.py", line 127, in load_model
model = load_quantized(model_name)
File "C:\Users\<user>\Downloads\oobabooga_windows\oobabooga_windows\text-generation-webui\modules\GPTQ_loader.py", line 193, in load_quantized
model = model.to(torch.device('cuda:0'))
File "C:\Users\<user>\Downloads\oobabooga_windows\oobabooga_windows\installer_files\env\lib\site-packages\transformers\modeling_utils.py", line 1896, in to
return super().to(*args, **kwargs)
File "C:\Users\<user>\Downloads\oobabooga_windows\oobabooga_windows\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1145, in to
return self._apply(convert)
File "C:\Users\<user>\Downloads\oobabooga_windows\oobabooga_windows\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
module._apply(fn)
File "C:\Users\<user>\Downloads\oobabooga_windows\oobabooga_windows\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
module._apply(fn)
File "C:\Users\<user>\Downloads\oobabooga_windows\oobabooga_windows\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 820, in _apply
param_applied = fn(param)
File "C:\Users\<user>\Downloads\oobabooga_windows\oobabooga_windows\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1143, in convert
return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
File "C:\Users\<user>\Downloads\oobabooga_windows\oobabooga_windows\installer_files\env\lib\site-packages\torch\cuda\__init__.py", line 239, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled
Done!
Press any key to continue . . .
`
System Info
Windows 11, NVIDIA GTX 4080
Same error for me. Tried to run the update. Also shows the CUDA EnvironmentError. raise EnvironmentError('CUDA_HOME environment variable is not set. ' OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
Same here trying to load the model : gpt-x-alpaca-13b-native-4bit-128g-cuda
installer_files\env\lib\site-packages\torch\cuda_init_.py", line 239, in _lazy_init raise AssertionError("Torch not compiled with CUDA enabled") AssertionError: Torch not compiled with CUDA enabled
I uninstalled Torch, and reinstalled the cuda toolkit. Then ran the update_windows batch file for oobabooga. And finally reinstalled torch.
That solved it... maybe overkill, but it worked.
If any of you can still reproduce this issue, I'd like to fix it. Please perform a fresh install and send the entire install log. There should be an error in there which will help me pinpoint the issue.
Same issue. During install it says "No CUDA runtime is found" and ends up with "CUDA kernal compilation failed"
Same issue. During install it says "No CUDA runtime is found" and ends up with "CUDA kernal compilation failed"
@Sinxain If you can send the entire install log, I can look into it.
Same issue. During install it says "No CUDA runtime is found" and ends up with "CUDA kernal compilation failed"
@Sinxain If you can send the entire install log, I can look into it.
@Sinxain thank you for this. I noticed for some reason the install is making references to a global Python and Pip install here c:\users\user\appdata\roaming\python\python310\site-packages when it should be using the Python install of the Miniconda environment. I think what's happening is that CUDA is being installed to the global system environment and is not accessible in the local Miniconda environment or vice versa.
The same error is happening here too https://github.com/oobabooga/one-click-installers/issues/26
I'll check it out
Getting same issue on M1 Mac Studio
Download current version of Macos one-click installer Install for M1 or CPU. Select any model. Attempt to run start_macos script
Tried deleting/reinstalling several times.
Getting same issue on M1 Mac Studio
Download current version of Macos one-click installer Install for M1 or CPU. Select any model. Attempt to run start_macos script
Tried deleting/reinstalling several times.
Same on M2 Mac
So I solved this issue on Windows by removing a bunch of duplicate/redundant python installations in my environment path. I left only miniconda, and the only way to access python is via activating a conda environment.
I also made sure to delete any extraneous site-package folders from the other python installs because for some reason it was still looking for pytorch in them.
Finally I installed torch via the conda install method listed on the cuda website: https://pytorch.org/get-started/locally/
For example, mine was: conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia with my RTX3080.
Hope this helps people, or maybe lends some ideas on the installer checking for these sorts of conflicts?
I've made some changes here which might fix this problem. If someone who still has this problem could check, I'd appreciate it
I've made some changes here which might fix this problem. If someone who still has this problem could check, I'd appreciate it
It's now loading the webui on localhost:7860 which is great progress. Thank you!
But getting a different error when actually running a query "RuntimeError: MPS does not support cumsum op with int64 input" but not sure if that's related. The UI briefly displays "The assistant is typing..." and then text output area goes blank.
I'll try my Mac Studio with a different model later.
2020 M1 Macbook Air 8GB RAM. Pythia-410M-deduped. M1 option selected in menu during installation.
Details of error: Gradio HTTP request redirected to localhost :) bin /Users/xxxxxxxxxx/Downloads/one-click-installers-main/installer_files/env/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so /Users/xxxxxxxxxx/Downloads/one-click-installers-main/installer_files/env/lib/python3.10/site-packages/bitsandbytes/cextension.py:33: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable. warn("The installed version of bitsandbytes was compiled without GPU support. " Loading EleutherAI_pythia-410m-deduped... Loaded the model in 2.14 seconds. Loading the extension "gallery"... Ok. Running on local URL: http://127.0.0.1:7860
To create a public link, set share=True in launch().
/Users/xxxxxxxxxx/Downloads/one-click-installers-main/installer_files/env/lib/python3.10/site-packages/transformers/generation/utils.py:690: UserWarning: MPS: no support for int64 repeats mask, casting it to int32 (Triggered internally at /Users/runner/work/_temp/anaconda/conda-bld/pytorch_1678454852765/work/aten/src/ATen/native/mps/operations/Repeat.mm:236.)
input_ids = input_ids.repeat_interleave(expand_size, dim=0)
Traceback (most recent call last):
File "/Users/xxxxxxxxxx/Downloads/one-click-installers-main/text-generation-webui/modules/callbacks.py", line 66, in gentask
ret = self.mfunc(callback=_callback, **self.kwargs)
File "/Users/xxxxxxxxxx/Downloads/one-click-installers-main/text-generation-webui/modules/text_generation.py", line 290, in generate_with_callback
shared.model.generate(**kwargs)
File "/Users/xxxxxxxxxx/Downloads/one-click-installers-main/installer_files/env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/Users/xxxxxxxxxx/Downloads/one-click-installers-main/installer_files/env/lib/python3.10/site-packages/transformers/generation/utils.py", line 1485, in generate
return self.sample(
File "/Users/xxxxxxxxxx/Downloads/one-click-installers-main/installer_files/env/lib/python3.10/site-packages/transformers/generation/utils.py", line 2521, in sample
model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
File "/Users/xxxxxxxxxx/Downloads/one-click-installers-main/installer_files/env/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py", line 710, in prepare_inputs_for_generation
position_ids = attention_mask.long().cumsum(-1) - 1
RuntimeError: MPS does not support cumsum op with int64 input
Output generated in 0.53 seconds (0.00 tokens/s, 0 tokens, context 38, seed 1967011890)
@greencat667 I don't know much about Pythia but it looks like other people have your error too. Any ggml or hf model should work though. I'm able to use https://huggingface.co/Drararara/llama-7b-ggml on my Intel-based Macbook for example.
Same on M2 Macbook! UI opens successfully. But a slightly different error when trying to input a text:
Users/{user}/Documents/AI/oobabooga/one-click-installers/installer_files/env/lib/python3.10/site-packages/transformers/generation/utils.py:690: UserWarning: MPS: no support for int64 repeats mask, casting it to int32 (Triggered internally at /Users/runner/work/_temp/anaconda/conda-bld/pytorch_1678454852765/work/aten/src/ATen/native/mps/operations/Repeat.mm:236.)
input_ids = input_ids.repeat_interleave(expand_size, dim=0)
Traceback (most recent call last):
File "/Users/{user}/Documents/AI/oobabooga/one-click-installers/text-generation-webui/modules/callbacks.py", line 66, in gentask
ret = self.mfunc(callback=_callback, **self.kwargs)
File "/Users/{user}/Documents/AI/oobabooga/one-click-installers/text-generation-webui/modules/text_generation.py", line 290, in generate_with_callback
shared.model.generate(**kwargs)
File "/Users/{user}/Documents/AI/oobabooga/one-click-installers/installer_files/env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/Users/{user}/Documents/AI/oobabooga/one-click-installers/installer_files/env/lib/python3.10/site-packages/transformers/generation/utils.py", line 1485, in generate
return self.sample(
File "/Users/{user}/Documents/AI/oobabooga/one-click-installers/installer_files/env/lib/python3.10/site-packages/transformers/generation/utils.py", line 2524, in sample
outputs = self(
File "/Users/{user}/Documents/AI/oobabooga/one-click-installers/installer_files/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/{user}/Documents/AI/oobabooga/one-click-installers/installer_files/env/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 938, in forward
outputs = self.model.decoder(
File "/Users/{user}/Documents/AI/oobabooga/one-click-installers/installer_files/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/{user}/Documents/AI/oobabooga/one-click-installers/installer_files/env/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 648, in forward
pos_embeds = self.embed_positions(attention_mask, past_key_values_length)
File "/Users/{user}/Documents/AI/oobabooga/one-click-installers/installer_files/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/{user}/Documents/AI/oobabooga/one-click-installers/installer_files/env/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 115, in forward
positions = (torch.cumsum(attention_mask, dim=1).type_as(attention_mask) * attention_mask).long() - 1
RuntimeError: MPS does not support cumsum op with int64 input
Output generated in 2.02 seconds (0.00 tokens/s, 0 tokens, context 34, seed 1652130967)
the most lightweight one: D) OPT 350M Now tying with the one you linked!
Update: This model works fine https://huggingface.co/Drararara/llama-7b-ggml
@Sinxain can you try a different model like https://huggingface.co/wcde/llama-7b-4bit-gr128?
Seems to be working now. I uninstalled python and deleted the python folder in AppData. Installed oobabooga using the one-click installer from the official page and its working. My guess that folder in AppData was the culprit.
Ignoring bitsandbytes: markers 'platform_system != "Windows"' don't match your environment Ignoring llama-cpp-python: markers 'platform_system != "Windows"' don't match your environment
Getting the Errors above when trying to install with this command: "install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia"
running on Windows 11 so I don´t know why it won´t work, or even uses env instead of venv like 90% of the other AI stuff out there...
@TripleKiller666 that is a warning, not an error. If you didn't receive an error during installation, there should be no problem with it and you should be set to run a model.
We won't be changing to venv anytime soon unless there is some benefit?
Dunno about any other benefit than it installing all the Cuda stuff properly on Windows. Even with the "fixed" installer Files I get these Errors:
No CUDA runtime is found, using CUDA_HOME='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8'
running install
G:\AI_STUFF\ooga2\one-click-installers-main\installer_files\env\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_ext
G:\AI_STUFF\ooga2\one-click-installers-main\installer_files\env\lib\site-packages\torch\utils\cpp_extension.py:358: UserWarning: Error checking compiler version for cl: [WinError 2] Das System kann die angegebene Datei nicht finden
warnings.warn(f'Error checking compiler version for {compiler}: {error}')
Traceback (most recent call last):
File "
note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure
× Encountered error while trying to install package. ╰─> quant-cuda
note: This is an issue with the package mentioned above, not pip. hint: See above for output from the failure. ERROR: GPTQ CUDA kernel compilation failed. Attempting installation with wheel. Collecting quant-cuda==0.0.0 Downloading https://github.com/jllllll/GPTQ-for-LLaMa-Wheels/raw/main/quant_cuda-0.0.0-cp310-cp310-win_amd64.whl (398 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 398.4/398.4 kB 3.5 MB/s eta 0:00:00 Installing collected packages: quant-cuda Successfully installed quant-cuda-0.0.0