Old CPU | AVX / AVX2 required to run privateGPT?

[Shutiri]

Hello,

Thank you so much for building this amazing project!

I went all the way through installation and ingestion, no problem. When running the command "python privateGPT.py", I get errors like these:

OSError: [WinError -1073741795] Windows Error 0xc000001d

I tried with both Vicuna13B and GPT4all, different error message but same result.

GPT4 says it could be because my CPU doesn't support AVX2. I'd like to confirm that before buying a new CPU for privateGPT :) !

Thank you!

---

My system:

Windows 10 Home Version 10.0.19045 Build 19045 Processor Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3401 Mhz, 4 Core(s), 8 Logical Processor(s) Installed Physical Memory (RAM) 16.0 GB Total Virtual Memory 31.9 GB

Python 3.10.6 64bit

[anantshri]

try switching to llama backend. i.e. instead of using gpt4all model use the other models from llama backend i have had better luck with them

[Shutiri]

Thank you for your answer. As stated, I tried with Vicuna 13B as well, this is a LlaMA model.

Do you know if privateGPT works with old processors that support AVX but not AVX2? Is that the issue?

[darrinh]

I tried vicuna and got:

Using embedded DuckDB with persistence: data will be stored in: db
gptj_model_load: loading model from 'models/wizard-vicuna-13B.ggmlv3.q4_1.bin' - please wait ...
gptj_model_load: invalid model file 'models/wizard-vicuna-13B.ggmlv3.q4_1.bin' (bad magic)

Not sure if I'm missing something or not.

[darrinh]

Also tried: mpt-7b-instruct.ggmlv3.q8_0.bin and get:

gptj_model_load: invalid model file 'models/mpt-7b-instruct.ggmlv3.q8_0.bin' (bad vocab size 2007 != 4096)

[anantshri]

1. ensure your models are quantized with latest version of llama.cpp they changed format recently. (19 may)
2. if you get bad magic that could be coz the quantized format is too new in which case pip install llama-cpp-python==0.1.53 would help

[anantshri]

Anyone else stuck in simmilar situation https://blog.anantshri.info/privategpt-and-cpus-with-no-avx2/ is what i did and has been able to run privategpt on avx only system.