AssertionError: Torch not compiled with CUDA enabled

[smartdawg]

using python virtual env on windows. used example, downloaded all but got the error

saved as: C:\Users....cache\huggingface\hub\models--v2ray--Llama-3-70B\snapshots...\splitted_model\lm_head.safetensors 100%| ... 83/83 [31:20<00:00, 22.65s/it] Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. new version of transfomer, no need to use BetterTransformer, try setting attn impl to sdpa... attn imp: <class 'transformers.models.llama.modeling_llama.LlamaSdpaAttention'> Traceback (most recent call last): File "", line 198, in run_module_as_main File "", line 88, in run_code File "C:\Users...\AirLLM\test.py", line 3, in model = AutoModel.from_pretrained("v2ray/Llama-3-70B") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users...\AirLLM.venv\Lib\site-packages\airllm\auto_model.py", line 54, in from_pretrained
return class(pretrained_model_name_or_path, *inputs, ** kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\Ric\Desktop\dev\AirLLM.venv\Lib\site-packages\airllm\airllm.py", line 9, in init super(AirLLMLlama2, self).init(*args, **kwargs) File "C:\Users...\AirLLM.venv\Lib\site-packages\airllm\airllm_base.py", line 127, in init self.init_model() File "C:\Users...\AirLLM.venv\Lib\site-packages\airllm\airllm_base.py", line 225, in init_model
set_module_tensor_to_device(self.model, buffer_name, self.running_device, value=buffer, File "C:\Users...\AirLLM.venv\Lib\site-packages\accelerate\utils\modeling.py", line 404, in set_module_tensor_to_device new_value = value.to(device) ^^^^^^^^^^^^^^^^ File "C:\Users...\AirLLM.venv\Lib\site-packages\torch\cuda_init.py", line 284, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled") AssertionError: Torch not compiled with CUDA enabled (.venv) PS C:\Users...\AirLLM> pip list Package Version

---

accelerate 0.32.0 aiohttp 3.9.5 aiosignal 1.3.1 airllm 2.8.3 attrs 23.2.0 certifi 2024.7.4 charset-normalizer 3.3.2 colorama 0.4.6 coloredlogs 15.0.1 datasets 2.20.0 dill 0.3.8 filelock 3.15.4 frozenlist 1.4.1 fsspec 2024.5.0 huggingface-hub 0.23.4 humanfriendly 10.0 idna 3.7 intel-openmp 2021.4.0 Jinja2 3.1.4 MarkupSafe 2.1.5 mkl 2021.4.0 mpmath 1.3.0 multidict 6.0.5 multiprocess 0.70.16 networkx 3.3 numpy 1.26.4 optimum 1.21.1 packaging 24.1 pandas 2.2.2 pip 24.0 protobuf 5.27.2 psutil 6.0.0 pyarrow 16.1.0 pyarrow-hotfix 0.6 pyreadline3 3.4.1 python-dateutil 2.9.0.post0 pytz 2024.1 PyYAML 6.0.1 regex 2024.5.15 requests 2.32.3 safetensors 0.4.3 scipy 1.14.0 sentencepiece 0.2.0 six 1.16.0 sympy 1.12.1 tbb 2021.13.0 tokenizers 0.19.1 torch 2.3.1 tqdm 4.66.4 transformers 4.42.3 typing_extensions 4.12.2 tzdata 2024.1 urllib3 2.2.2 xxhash 3.4.1 yarl 1.9.4

[SueCripter]

I suspect that the problem is because your version of Torch was not compiled with CUDA enabled.

There are instructions on this page on how to get Torch with the CUDA version that you have: https://pytorch.org/get-started/locally

[gokulcoder7]

you need to download cuda toolkit , https://medium.com/@lucsiecker/how-i-enable-my-cuda-capable-graphics-card-for-machine-learning-windows-10-11-b27a4beee326