UnboundLocalError: cannot access local variable 'llm' where it is not associated with a value

[QuantumPickleJar]

Describe the bug and how to reproduce it I was able to product this bug by trying to load up the Vicuna LLAMA model from Huggingface. Here's the link to the model I attempted to use: https://huggingface.co/eachadea/ggml-vicuna-13b-1.1

Expected behavior Upon executing python privateGPT.py, I should be prompted to enter a query, but instead a failure with the following console output is received:

Using embedded DuckDB with persistence: data will be stored in: db Model llama not supported! Traceback (most recent call last): File "X:\git\privategpt\privateGPT\privateGPT.py", line 75, in main() File "X:\git\privategpt\privateGPT\privateGPT.py", line 39, in main qa = RetrievalQA.from_chain_type(llm=llm, chain_type="stuff", retriever=retriever, return_source_documents= not args.hide_source) ^^^ UnboundLocalError: cannot access local variable 'llm' where it is not associated with a value

Environment (please complete the following information):

• OS / hardware: Win10; 2017 Asus ROG Laptop
• Python version 3.11.3

I know that Vicuna is a LLAMA based model, so I'm not sure why the interpreter is saying it's an unsupported model type.
Let me know if there's anything else I can include to help resolve this: this is my first issue I've posted on GitHub. Thanks for your patience!

[PulpCattel]

Model llama not supported!

This is because of a small bug there's in the code. We don't correctly terminate when an invalid model_type is found.

Check your .env config file, you somehow have MODEL_TYPE=llama but should be MODEL_TYPE=LlamaCpp or MODEL_TYPE=GPT4All

[QuantumPickleJar]

Ah, okay: I hadn't realized the LlamaCpp was the explicit modeltype. Thanks! Adjusting the MODEL_TYPE to be LlamaCpp results in the following new value_error: Using embedded DuckDB with persistence: data will be stored in: db llama.cpp: loading model from X:\models\ggml-vicuna-13b-1.1-q4_2.bin error loading model: unrecognized tensor type 4

llama_init_from_file: failed to load model Traceback (most recent call last): File "X:\git\privategpt\privateGPT\privateGPT.py", line 75, in main() File "X:\git\privategpt\privateGPT\privateGPT.py", line 33, in main llm = LlamaCpp(model_path=model_path, n_ctx=model_n_ctx, callbacks=callbacks, verbose=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "pydantic\main.py", line 341, in pydantic.main.BaseModel.init pydantic.error_wrappers.ValidationError: 1 validation error for LlamaCpp root Could not load Llama model from path: X:\models\ggml-vicuna-13b-1.1-q4_2.bin. Received error (type=value_error)

I've also tried reversing slash directions in the MODEL_PATH but that didn't seem to alter the error received in any way: not even the line numbers were different. Hoping it's just more user error obstructing expected functionality!

[maozdemir]

use Q4_0 models.

[QuantumPickleJar]

Downloading one now. I'm curious what difference this makes: is there some kind of naming convention being followed that I'm simple not aware of yet? I'd love to know what this is or if there's further reading I could look into.

[QuantumPickleJar]

It got a little further using a model ending in q4_0, it looks like it was at least able to partially load some of the model functions. Here's the output and error recieved upon updating the MODEL_PATH to reflect the newly download q4_0 model:

$ python privateGPT.py Using embedded DuckDB with persistence: data will be stored in: db llama.cpp: loading model from X:/models/Vicuna/ggml-old-vic13b-q4_0.bin llama_model_load_internal: format = ggjt v1 (pre #1405) llama_model_load_internal: n_vocab = 32000 llama_model_load_internal: n_ctx = 1000 llama_model_load_internal: n_embd = 5120 llama_model_load_internal: n_mult = 256 llama_model_load_internal: n_head = 40 llama_model_load_internal: n_layer = 40 llama_model_load_internal: n_rot = 128 llama_model_load_internal: ftype = 4 (mostly Q4_1, some F16) llama_model_load_internal: n_ff = 13824 llama_model_load_internal: n_parts = 1 llama_model_load_internal: model size = 13B error loading model: this format is no longer supported (see https://github.com/ggerganov/llama.cpp/pull/1305) llama_init_from_file: failed to load model Traceback (most recent call last): File "X:\git\privategpt\privateGPT\privateGPT.py", line 75, in main() File "X:\git\privategpt\privateGPT\privateGPT.py", line 33, in main llm = LlamaCpp(model_path=model_path, n_ctx=model_n_ctx, callbacks=callbacks, verbose=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "pydantic\main.py", line 341, in pydantic.main.BaseModel.init pydantic.error_wrappers.ValidationError: 1 validation error for LlamaCpp root Could not load Llama model from path: X:/models/Vicuna/ggml-old-vic13b-q4_0.bin. Received error (type=value_error)