Java API 1.1.5 Doesn't Load New GGUF Models

[seleniumRnD]

System Info

Official Java API Doesn't Load GGUF Models

GPT4All 2.5.2 now requires the new GGUF model format, but the Official API 1.1.5 has not been updated and ONLY works with the previous GLLML bin models. The Official Java API should track the main app release and have the ability to work with the same file format (now GGUF).

`java.lang.IllegalStateException: Could not load, gpt4all backend returned error: Model format not supported (no matching implementation found)

Information

The API supports an older version of the app:

'com.hexadevlabs:gpt4all-java-binding:1.1.5' INFO com.hexadevlabs.gpt4all.LLModel - Java bindings for gpt4all version: 2.4.11

Reproduction

Steps to reproduce:

new LLModel(Path.of(Config.modelPath("C:/opt/gpt4all-models/wizardlm-13b-v1.2.Q4_0.gguf")

Expected behavior

The Official Java API should track the main app release and have the ability to work with the same file format as the app (GGUF).

[cebtenzzre]

cc @lordofthejars @felix-zaslavskiy

[Onkitova]

+1 Same for #csharp bindings.

[cebtenzzre]

Same for #csharp bindings.

#1651 fixes the C# bindings.

[ygoron]

Having the same issue with Java binding

[jefrajames]

+1 Quarkus example no longer working: https://developers.redhat.com/articles/2023/09/25/integrate-your-quarkus-application-gpt4all#

Workaround to make Quarkus example work: download Falcon model in bin format (ggml-model-gpt4all-falcon-q4_0.bin) from https://huggingface.co/nomic-ai/gpt4all-falcon-ggml/tree/main. Copy it somewhere and change your application.properties accordingly. For instance:

model.file=ggml-model-gpt4all-falcon-q4_0.bin model.path=/Users/powerj/javadev/lab/gpt4all/models

The problem is that GPT4All Java binding supports GPT4All version 2.4.11. Current version is 2.5.4 and GGUF files are required since 2.5.2.

[javaaiorg]

Thanks for @jefrajames 's information.

I have the same propblem, and I find out how to solve it.

1. Download GPT4All client application and install it: https://gpt4all.io/index.html

2. After installed, you will got the newest libs (v 2.5.4) that supports GGUF format models in client install directory: gpt4all/lib.

3. Then change the Java code, custom GPT4All dynamic link libarary path by the following code: LLModel.LIBRARY_SEARCH_PATH = "C:\\Users\\?changeMe\\gpt4all\\lib\\";

After that, my application works fine.

Tests: My modules: nous-hermes-llama2-13b.Q4_0.gguf My prompt: "### Human:\nWhat is the meaning of life\n### Assistant:" GPT LLaMa response: The meaning of life is a philosophical and personal question that has been debated for centuries. It is a subjective question, and the answer can vary from person to person. Some people believe that the meaning of life is to find happiness, others believe it's about fulfilling one's purpose or destiny, and still others believe it's about making a positive impact on the world. Ultimately, the meaning of life is a personal journey that each individual must explore and discover for themselves.

[floscher]

@javaaiorg I can confirm that this works. In my case with the https://huggingface.co/TheBloke/em_german_mistral_v01-GGUF model (I had issues with the Q4_0 variant, but Q3_K_M works) on Ubuntu 22.04.1 :tada:

@felix-zaslavskiy Is there anything that people could help with towards releasing a version 1.1.6? It would be great if the package in MavenCentral would be updated, even if it's just that the native libs are bumped to 2.5.4.

[felix-zaslavskiy]

@javaaiorg I can confirm that this works. In my case with the https://huggingface.co/TheBloke/em_german_mistral_v01-GGUF model (I had issues with the Q4_0 variant, but Q3_K_M works) on Ubuntu 22.04.1 🎉

@felix-zaslavskiy Is there anything that people could help with towards releasing a version 1.1.6? It would be great if the package in MavenCentral would be updated, even if it's just that the native libs are bumped to 2.5.4.

Yea it's on me. It's hard to work on these projects when life is busy. I'll work to get an updated version out!

[ida-greuelr]

I would also highly appreciate an updated bindings package. I did the libs "hack" but the result was pending from working flawlessly to segfaulting the whole surrounding container. Thanks for working on this. If we can help in testing new commits on master or something else, let me know.

[ThiloteE]

Java, Go and C# bindings were removed by commit beaede03fba996071a4b549107bcfa2b8a4066bf.

Java bindings for llama.cpp (which a large part of GPT4All is built upon) can be tracked at https://github.com/kherud/java-llama.cpp, but I am sure GPT4All has some peculiarities that would require somebody to keep track of. Simply put, to bring Java bindings back, we would need a maintainer.

[felix-zaslavskiy]

Java, Go and C# bindings were removed by commit beaede0.

Java bindings for llama.cpp (which a large part of GPT4All is built upon) can be tracked at https://github.com/kherud/java-llama.cpp, but I am sure GPT4All has some peculiarities that would require somebody to keep track of. Simply put, to bring Java bindings back, we would need a maintainer.

Hi I wrote the original Java binding but unfortunately I don't have time right now to maintain it due to personal reasons. I may update it but on my own repo in future without the requirement of it having to be integrated into the main project. I appreciate all the feedback and I know some folks still want to use it but I just don't have time to maintain it right now.