Support local models.json file for chat templates (for python bindings, and in general)

[RossBencina]

Feature request

Support loading models.json from well known local location(s), such as:

• The model directory specified when instantiating GPT4All (and perhaps also its parent directories)
• The default location used by the GPT4All application.

Additionally, the GPT4All application could place a copy of models.json in the same directory as the downloaded model file.

@cosmic-snow pointed out that the application may store models in subdirectories of the models directory, so it may not be 100% clear that there is a single unique "application model directory" in which to search for models.json -- or the model directory passed to the python API may in fact be a subdirectory of the "application model directory".

Motivation

Right now, the python binding only accesses models.json by loading it from the web, and only if allow_download is True. Here's the code:

https://github.com/nomic-ai/gpt4all/blob/a63093554fb5098678eda94115b49754fa25c35e/gpt4all-bindings/python/gpt4all/gpt4all.py#L130-L140

This is a problem because it means that there is no way to access the models.json prompt templates without setting allow_download to True, and incurring the delay (and errors) of a network transaction, to use what is supposed to be a local model.

Further motivation: for ease of use, I would like my users to be able to use GPT4All Application to manage and test model downloads, but to also use these models from my python code. In order to to use the GPT4All chat completions API in my python code, I need to have working prompt templates. For local use I do not want my python code to set allow_download = True. Therefore I need the GPT4All python bindings to access a local model.json -- ideally one automatically downloaded by the GPT4All application.

Related: #1241

Your contribution

I could potentially submit a PR but I would like to know that it has a chance at being acceptable before doing work in that direction.

Possible PRs to Python bindings:

• Allow optionally passing a path to a local models.json when instantiating GPT4ALL
• Search for models.json in model directory and its parents
• Search for models.json in GPT4All application directories (e.g. `C:\Users\Ross\AppData\Roaming\nomic.ai\models.json') but someone would need to advise the best way to find these directories on each platform.
• If allow_download is True, store models.json along side the downloaded model.

Possible PRs to Application:

• Make downloader save models.json in same directory as model (ideally, merge entries if an existing models.json is present)

[niansa]

This is part of #1241

[niansa]

Closing this as duplicate.

If you need anything else, please feel free to open more issues!

[cosmic-snow]

It's not just part of #1241 😑

[niansa]

From my understanding, this would simply be a local model repository. But if you think otherwise, I apologize! Didn't mean to suppress your ideas :slightly_smiling_face:

[AndriyMulyar]

This is a great feature and the current functionality is non-ideal. Happy to accept modifications that allow:

• models.json to download once and then get cached in the gpt4all cache directory.
• if there is network available, we should check if there is a new version of models.json present (otherwise, the bindings would never update the model list when new models are added)

[RossBencina]

Happy to accept modifications that allow:

• models.json to download once and then get cached in the gpt4all cache directory.

Is this the same directory that the gpt4all app stores models.json? If so, what is the path to this directory on each target platform? I have seen ~/.cache mentioned but on my Windows machine models.json is located in C:\Users\Ross\AppData\Roaming\nomic.ai

[cosmic-snow]

Happy to accept modifications that allow:

• models.json to download once and then get cached in the gpt4all cache directory.

Is this the same directory that the gpt4all app stores models.json? If so, what is the path to this directory on each target platform? I have seen ~/.cache mentioned but on my Windows machine models.json is located in C:\Users\Ross\AppData\Roaming\nomic.ai

• Python bindings by default put everything into ~/.cache/gpt4all on all platforms.

• The chat GUI puts things into folders that are preset through the Qt APIs. So that's different on every platform and can also be different depending on the type: e.g. on Windows the settings are in ~\AppData\Roaming\nomic.ai\GPT4All and the default model folder is in Local instead.

• The bindings interact with the backend through a C API, but the backend itself does not have a dependency on Qt, so it's not possible to expose those Qt paths through the backend.

Also see: the Python bindings docs have an entry regarding the models folder.

[AndriyMulyar]

@niansa can you own organizing this across all bindings authors and @manyoso

[dlippold]

Because the main issue in the above discussion seems to be that python should have access to the file models.json, in particular when not internet connection is available, I want to add another motivation for an local file like models.json.

When I download a new model e.g. from TheBloke (https://huggingface.co/TheBloke) I want to provide information about that model to all user of the GUI, e.g. the informations system prompt and prompt template and (when the new release is out with the fix of #1749) the context length. That is currently not possible but would be possible by providing an additionally local file like models.json (e.g. models-add.json).