When i run privateGPT.py I got an error that says: cannot access local variable 'llm' where it is not associated with a value

[busyneuron]

Error running the privateGPT.py file on cmd. Here is the full error:

---

Using embedded DuckDB with persistence: data will be stored in: db Model GPT4ALL not supported! Traceback (most recent call last): File "C:\Users\soyel\Desktop\privateGpt\privateGPT.py", line 77, in main() File "C:\Users\soyel\Desktop\privateGpt\privateGPT.py", line 41, in main qa = RetrievalQA.from_chain_type(llm=llm, chain_type="stuff", retriever=retriever, return_source_documents= not args.hide_source) ^^^ UnboundLocalError: cannot access local variable 'llm' where it is not associated with a value

---

I think it should load everything fine.

• Windows 10
• Python 3.11.3

[GianlucaMattei]

It can not retrieve your model. You should add to your comment the .env file to understand what is happening. Anyway I can say there is at least one error. The model (MODEL_TYPE) is named GTP4All not GTP4ALL. Try to edit this in your .env file first of all

[busyneuron]

PERSIST_DIRECTORY=db MODEL_TYPE=GPT4All MODEL_PATH=models/ggml-gpt4all-j-v1.3-groovy.bin EMBEDDINGS_MODEL_NAME=all-MiniLM-L6-v2 MODEL_N_CTX=1000 TARGET_SOURCE_CHUNKS=4

[busyneuron]

Okay it is solved, now i can enter a query, but it lasts too long to answer, is it because of my computer's performance?

[GianlucaMattei]

Okay it is solved, now i can enter a query, but it lasts too long to answer, is it because of my computer's performance?

Don't know, I have a 3060m and it takes a lot to answer. I was wondering your same question since I pretty new to these things

[johnbrisbin]

Currently, the computer's CPU is the only resource used. GPU support is on the way, but getting it installed is tricky. It helps greatly with the ingest, but I have not yet seen improvement on the same scale with the query side, but the installed GPU only has about 5.5GB free for model layers.

[GianlucaMattei]

Currently, the computer's CPU is the only resource used. GPU support is on the way, but getting it installed is tricky. It helps greatly with the ingest, but I have not yet seen improvement on the same scale with the query side, but the installed GPU only has about 5.5GB free for model layers.

I thought each model uses the GPU Thus the *.bin models use the CPU? Or which type of models use it?

[johnbrisbin]

@GianlucaMattei, Virtually every model can use the GPU, but they normally require configuration to use the GPU. In privateGPT we cannot assume that the users have a suitable GPU to use for AI purposes and all the initial work was based on providing a CPU only local solution with the broadest possible base of support. CPU only models are dancing bears. It's not how well the bear dances, it's that it dances at all. Nonetheless, using CPU only models is a valuable proof of concept for purely local and private GPT. Now, enabling GPU functionality is a pretty high priority but it requires getting all the supporting software downloaded and installed in the user environment. That includes system modules like the CUDA libraries that all software to connect with a GPU in a standard way. It also requires modules added and upgraded in the python modules used by privateGPT to access the CUDA system libraries. Some of these have to be recompiled to support new functionality and that requires installing compilers on the user machine to perform the task... As you can see it gets complicated, largely because we have to assume that the user does not have all the stuff installed that the developers have collected over time. Once all that is in place, enabling GPU usage in a given model is usually as simple as adding a parameter on the command line with all but a few models. Unfortunately, exactly what parameter you need to add to enable GPU access is not standardized and various models use different methods and have additional tweaks to fine tune the GPU interface. The last part is pretty important because most large GPU models will not fit and run in common retail Nvidia video card memory. Relatively few will fit fully and run on a RTX 4090 with 24GB. So you need tricks to make it work. And it goes on...