AssertionError: Torch not compiled with CUDA enabled

[kang2000]

Anyone know what is the issue causing this error? ===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

CUDA SETUP: Required library version not found: libsbitsandbytes_cpu.so. Maybe you need to compile it from source? CUDA SETUP: Defaulting to libbitsandbytes_cpu.so... argument of type 'WindowsPath' is not iterable C:\Users\Kang\miniconda3\envs\minigpt4\lib\site-packages\bitsandbytes\cextension.py:31: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable. warn("The installed version of bitsandbytes was compiled without GPU support. " Loading checkpoint shards: 0%| | 0/3 [00:04<?, ?it/s] Traceback (most recent call last): File "D:\llama\vicuna\MiniGPT-4\demo.py", line 60, in model = model_cls.from_config(model_config).to('cuda:{}'.format(args.gpu_id)) File "D:\llama\vicuna\MiniGPT-4\minigpt4\models\mini_gpt4.py", line 243, in from_config model = cls( File "D:\llama\vicuna\MiniGPT-4\minigpt4\models\mini_gpt4.py", line 90, in init self.llama_model = LlamaForCausalLM.from_pretrained( File "C:\Users\Kang\miniconda3\envs\minigpt4\lib\site-packages\transformers\modeling_utils.py", line 2795, in from_pretrained ) = cls._load_pretrained_model( File "C:\Users\Kang\miniconda3\envs\minigpt4\lib\site-packages\transformers\modeling_utils.py", line 3123, in _load_pretrained_model new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model( File "C:\Users\Kang\miniconda3\envs\minigpt4\lib\site-packages\transformers\modeling_utils.py", line 706, in load_state_dict_into_meta_model set_module_8bit_tensor_to_device( File "C:\Users\Kang\miniconda3\envs\minigpt4\lib\site-packages\transformers\utils\bitsandbytes.py", line 87, in set_module_8bit_tensor_to_device new_value = value.to(device) File "C:\Users\Kang\miniconda3\envs\minigpt4\lib\site-packages\torch\cuda_init.py", line 239, in _lazy_init raise AssertionError("Torch not compiled with CUDA enabled") AssertionError: Torch not compiled with CUDA enabled

[snutp]

Also having the same issue.

[kriszhangk]

Also having the same issue.

[zxcvbn114514]

+1

[5118Python]

+1

[Websteria]

Same here.

[zxcvbn114514]

I might have a solution: Check your torch version in the environment , it might be cpu version.If it is, install a new one.

[zxcvbn114514]

Solution 2: In mingpt4 environment ,uninstall the oringinal bitsandbytes and install this versionhttps://github.com/Keith-Hon/bitsandbytes-windows.It's designed for windows.

[kang2000]

I might have a solution: Check your torch version in the environment , it might be cpu version.If it is, install a new one.

Window 11 cpu: amd 5600x GPU: RTX3090

Im using torch2.0 which version are you using? (minigpt4) D:\llama\vicuna\MiniGPT-4>pip show torch Name: torch Version: 2.0.0 Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration Home-page: https://pytorch.org/ Author: PyTorch Team Author-email: [email protected] License: BSD-3 Location: c:\users\kang\miniconda3\envs\minigpt4\lib\site-packages Requires: filelock, jinja2, networkx, sympy, typing-extensions Required-by: accelerate, peft, sentence-transformers, timm, torchaudio, torchvision

[zxcvbn114514]

I might have a solution: Check your torch version in the environment , it might be cpu version.If it is, install a new one.

Window 11 cpu: amd 5600x GPU: RTX3090

Im using torch2.0 which version are you using? (minigpt4) D:\llama\vicuna\MiniGPT-4>pip show torch Name: torch Version: 2.0.0 Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration Home-page: https://pytorch.org/ Author: PyTorch Team Author-email: [email protected] License: BSD-3 Location: c:\users\kang\miniconda3\envs\minigpt4\lib\site-packages Requires: filelock, jinja2, networkx, sympy, typing-extensions Required-by: accelerate, peft, sentence-transformers, timm, torchaudio, torchvision

try solution2

[kang2000]

Guys i found the solution for this error. it is most probably we installed the wrong version of pytorch make sure when you do pip show torch it show version: 2.0.0+cu117

(minigpt4) D:\llama\vicuna\MiniGPT-4>pip show torch Name: torch Version: 2.0.0+cu117 Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration Home-page: https://pytorch.org/ Author: PyTorch Team Author-email: [email protected] License: BSD-3 Location: c:\users\kang\miniconda3\envs\minigpt4\lib\site-packages Requires: filelock, jinja2, networkx, sympy, typing-extensions Required-by: accelerate, peft, sentence-transformers, timm, torchaudio, torchvision

After that we need to edit the bitsandbytes files follow this reddit https://www.reddit.com/r/MachineLearning/comments/11kwdu9/d_tutorial_run_llama_on_8gb_vram_on_windows/