Maintain a list of supported models (if possible)

[gaurav-cointab]

If possible can you maintain a list of supported models.

If people can also list down which models have they been able to make it work, then it will be helpful. If yes, then with what settings.

[haris525]

working so far sentence-transformers/all-MiniLM-L6-v2 sentence-transformers/all-MiniLM-L12-v2

[recursionbane]

Working out of the box:

• ggml-gpt4all-j-v1.3-groovy.bin

Working after changing backend='llama' on line 30 in privateGPT.py, thanks to @PulpCattel:

• ggml-vicuna-13b-1.1-q4_2.bin
• ggml-gpt4all-l13b-snoozy.bin

Fails due to "bad magic":

• ggml-mpt-7b-instruct.bin

[PulpCattel]

@recursionbane ggml-gpt4all-l13b-snoozy.bin works if you change line 30 in privateGPT.py

Change this line llm = GPT4All(model=model_path, n_ctx=model_n_ctx, backend='gptj', callbacks=callbacks, verbose=False) to llm = GPT4All(model=model_path, n_ctx=model_n_ctx, backend='llama', callbacks=callbacks, verbose=False)

I.e., change the backend='gptj' part to llama. (I guess it could be made a config). See https://python.langchain.com/en/latest/modules/models/llms/integrations/gpt4all.html and https://docs.gpt4all.io/gpt4all_python.html#quickstart for more details.

---

Alternatively, you can set MODEL_TYPE=LlamaCpp in your .env config file (but this is much slower on my machine). I haven't tested ggml-vicuna-13b-1.1-q4_2.bin, but the q5 version works if you update langchain to the latest version (so I think your version should just work).

[lilinwang]

@recursionbane ggml-gpt4all-l13b-snoozy.bin works if you change line 30 in privateGPT.py

Change this line llm = GPT4All(model=model_path, n_ctx=model_n_ctx, backend='gptj', callbacks=callbacks, verbose=False) to llm = GPT4All(model=model_path, n_ctx=model_n_ctx, backend='llama', callbacks=callbacks, verbose=False)

I.e., change the backend='gptj' part to llama. (I guess it could be made a config). See https://python.langchain.com/en/latest/modules/models/llms/integrations/gpt4all.html and https://docs.gpt4all.io/gpt4all_python.html#quickstart for more details.

Alternatively, you can set MODEL_TYPE=LlamaCpp in your .env config file (but this is much slower on my machine). I haven't tested ggml-vicuna-13b-1.1-q4_2.bin, but the q5 version works if you update langchain to the latest version (so I think your version should just work).

Got the following error: privateGPT.py", line 35, in main llm = GPT4All(model=model_path, n_ctx=model_n_ctx, backend='llama', callbacks=callbacks, verbose=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "pydantic/main.py", line 341, in pydantic.main.BaseModel.init pydantic.error_wrappers.ValidationError: 1 validation error for GPT4All root Model.init() got an unexpected keyword argument 'n_parts' (type=type_error) Exception ignored in: <function Model.del at 0x293b66160> Traceback (most recent call last): File "/opt/homebrew/lib/python3.11/site-packages/pyllamacpp/model.py", line 402, in del if self._ctx: ^^^^^^^^^ AttributeError: 'GPT4All' object has no attribute '_ctx'

[PulpCattel]

@lilinwang To be clear, you only changed backend= right? And can you confirm the default configuration works for you with the default model?

You can always remove that change and try the MODEL_TYPE=LlamaCpp approach instead.

[deniskristak]

@recursionbane ggml-gpt4all-l13b-snoozy.bin works if you change line 30 in privateGPT.py

Change this line llm = GPT4All(model=model_path, n_ctx=model_n_ctx, backend='gptj', callbacks=callbacks, verbose=False) to llm = GPT4All(model=model_path, n_ctx=model_n_ctx, backend='llama', callbacks=callbacks, verbose=False)

I.e., change the backend='gptj' part to llama. (I guess it could be made a config). See https://python.langchain.com/en/latest/modules/models/llms/integrations/gpt4all.html and https://docs.gpt4all.io/gpt4all_python.html#quickstart for more details.

Alternatively, you can set MODEL_TYPE=LlamaCpp in your .env config file (but this is much slower on my machine). I haven't tested ggml-vicuna-13b-1.1-q4_2.bin, but the q5 version works if you update langchain to the latest version (so I think your version should just work).

I can confirm that leaving MODEL_TYPE=GPT4All and rewriting line 30 work for me :) thanks

[flikkr]

Got the following error: privateGPT.py", line 35, in main llm = GPT4All(model=model_path, n_ctx=model_n_ctx, backend='llama', callbacks=callbacks, verbose=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "pydantic/main.py", line 341, in pydantic.main.BaseModel.init pydantic.error_wrappers.ValidationError: 1 validation error for GPT4All root Model.init() got an unexpected keyword argument 'n_parts' (type=type_error) Exception ignored in: <function Model.del at 0x293b66160> Traceback (most recent call last): File "/opt/homebrew/lib/python3.11/site-packages/pyllamacpp/model.py", line 402, in del if self._ctx: ^^^^^^^^^ AttributeError: 'GPT4All' object has no attribute '_ctx'

This is caused by a broken dependency from pyllamacpp since they have changed their API. I fixed this by adding this line to my requirements.txt and then rerunning pip install -r requirements.txt

pyllamacpp==2.1.3

[andreascful]

SOLVED: It works when installing llama-cpp-python version 0.1.54. Interestingly it did not run with the newest version (0.1.56).

Whenever I try to use ggml-vic13b-uncensored-q5_0.bin or GPT4All-13B-snoozy.ggmlv3.q5_0.bin I receive the following error: error loading model: unknown (magic, version) combination: 67676a74, 00000002; is this really a GGML file? llama_init_from_file: failed to load model

I know that this is related to a recent update in llama.cpp but I am not sure how to fix it. Even using (and installing) the most recent versions of langchain and llama-cpp-python in the requirements.txt doesn't fix it.

@PulpCattel Which version of langchain are you using? Did you update any other packages in the requirements.txt?

[pseudotensor]

That error is because llama.cpp recently updated to version 3 quantization, all old models don't work. If you still want to use version 2 it's possible but you'll get left behind.

I recommend using only version 3 compatible models. We make this easy for the best wizardlm based models in h2oGPT: https://github.com/h2oai/h2ogpt#cpu

[ProxyAyush]

Working out of the box:

• ggml-gpt4all-j-v1.3-groovy.bin

Working after changing backend='llama' on line 30 in privateGPT.py, thanks to @PulpCattel:

• ggml-vicuna-13b-1.1-q4_2.bin
• ggml-gpt4all-l13b-snoozy.bin

Fails due to "bad magic":

• ggml-mpt-7b-instruct.bin

hey, ggml-mpt-7b works if you change line 36 in privateGPT.py file from 'gptj' to 'mpt'

[chitteshwari]

The ggml-vicuna-13b-1.1-q4_2.bin model does not get detected by privateGPT.py for me. I tried the solutions mentioned in this thread but none seem to work for me. During runtime, line 27 and 28 of privateGPT.py get skipped for me in spite of changing the MODEL_TYPE example.env to LlamaCpp and directly gives the following error at line 30:

ValidationError: 1 validation error for GPT4All root Failed to retrieve model (type=value_error)

Does anyone know how to solve this?