Set location of cached models and enable local loading

[ghost]

I have several gguf models stored locally in the project directory. Instead of loading from a url, I was trying to provide a relative path to the models. Something similar to what is possible in transformers.js where you can set the env variables as follows:

env.cacheDir = './src/models' env.allowRemoteModels = false; env.localModelPath = './src/models';

Is this possible with wllama?

Thanks - GREAT project!

[felladrin]

As far as I know, currently it's not possible to load files from the file-system.

You can import a file (using the HTML File Input element), though.

But even it was possible, there's a limitation of the file having to be smaller than <2 GB. If you have a file that is larger than that, you need to split the file using the gguf-split tool, as described in the Readme.

When it's split, I guess you won't be able to load it with the file-input element. In this case, the only way to run it is hosting it somewhere where it can be downloaded from. Most people host it on Hugging Face.

Are the files you have locally larger than 2 GB? If not, you can quickly launch a static server (e.g. python3 -m http.server 8000) and point your app to load a file from http://localhost:8000/<gguf_filename>.

[ngxson]

IIRC the env.localModelPath allow transformers.js to use models hosted on your own domain (instead of https://huggingface.co), but it does NOT mean you can load a local file from your computer (it's browser's limitation anyway)

For wllama, you are free to use model hosted from anywhere you want. loadModelFromUrl should allow you to do that:

wllama.loadModelFromUrl('./models/file.gguf')