[Windows] Exception when trying to load a model

[Holpak]

trafficstars

I try to run the sample code and get an error. It doesn't depend on the model I'm using.

Code:

from llama_cpp import Llama
llm = Llama(model_path="ggml-model.bin")
output = llm("Q: Name the planets in the solar system? A: ", max_tokens=32, stop=["Q:", "\n"], echo=True)
print(output)

Output:

llama.cpp: loading model from ggml-model.bin
Traceback (most recent call last):
  File "D:\Projects\llama-cpp-python-test\main.py", line 2, in <module>
    llm = Llama(model_path="ggml-model.bin")
  File "D:\Program Files\Python310\lib\site-packages\llama_cpp\llama.py", line 107, in __init__
    self.ctx = llama_cpp.llama_init_from_file(
  File "D:\Program Files\Python310\lib\site-packages\llama_cpp\llama_cpp.py", line 152, in llama_init_from_file
    return _lib.llama_init_from_file(path_model, params)
OSError: [WinError -1073741795] Windows Error 0xc000001d
Exception ignored in: <function Llama.__del__ at 0x000002595C895BD0>
Traceback (most recent call last):
  File "D:\Program Files\Python310\lib\site-packages\llama_cpp\llama.py", line 785, in __del__
    if self.ctx is not None:
AttributeError: 'Llama' object has no attribute 'ctx'

[jmtatsch]

Seems like loading the model already fails. Double check your model path.

[Holpak]

Path to the model is absolutely correct.

[abbiepup]

I have the same issue, path is also correct. Tried on Ubuntu and it works there, I only encounter this on windows.

from llama_cpp import Llama
llm = Llama(model_path="./models/7B/ggml-vicuna-7b-1.1-q4_0.bin")
output = llm("Q: Name the planets in the solar system? A: ", max_tokens=32, stop=["Q:", "\n"], echo=True)
print(output)

[mdawid]

@Holpak @KiraCoding Windows Error 0xc000001d occurs when the operating system detects an illegal or invalid instruction in a program. Python package is compiled from source with default llama.cpp flags LLAMA_AVX: ON and LLAMA_AVX2: ON If you try run the package with older processor llama.cpp will execute AVX2 instruction. I have to install this package with AVX2 flag disabled and it works fine.

first you have to clean previous installation:

pip uninstall llama-cpp-python
pip cache purge

then run CMAKE_ARGS="-DLLAMA_AVX2=off" FORCE_CMAKE=1 pip install llama-cpp-python

[abbiepup]

@mdawid I do have a pretty old CPU which doesn't support AVX2 so I guess my Ubuntu WSL did some magic here.

Thank you very much!