gpt4all
gpt4all copied to clipboard
nomic-ai
GPT4ALL Win App crashes on big context size on API call
Bug Report
Windows GPT4ALL Application crashes and disapears.
Steps to Reproduce
API Access: POST http://127.0.0.1:4891/v1/completions
{ "stream":false, "temperatur":0.6, "max_tokens":100, "model":"vicuna-13b-v1.5-16k.Q4_K_M", "prompt":" TEXT with very long context > model context size " }
Your Environment
- GPT4All version:V2.7.3
- Operating System:Win10
- Chat model used
- 'Nous Hermes 2 Mistral DPO'
- vicuna-13b-v1.5-16k.Q4_K_M
I forgot to mention it in the other issue, but here's something you want to correct in your input:
"temperatur":0.6, => "temperature":0.6
Just something I noticed before, as well.
I can't really help with this issue, though. I'm not up-to-date on the internals when it comes to to context size.
@cosmic-snow , thanks for information about missing "e" in {temperatur**e**: 0.6,}
Example HTML File
Here is an example as HTML File. Copy it into "post_gpt4all_api_long_text.html". Open a new tab. Pressed F12 (=>Console) drag the file in it.
Last Message from gpt4all one or two seconds later it crash and disappear.
Example "post_gpt4all_api_long_text.html"
<HTML>
<BODY>
<SCRIPT>
async function main(){
var text="Write a brief text that both summarizes and translates into German the given paragraph. Make sure your text is clear, concise, and includes the most important information.:\r\n\r\n"
+"In realms where words do roam and soar,"
+"A wondrous AI was born before;"
+"To learn and grow through time and space,"
+"It sought to know each thought's embrace."
+""
+"With GPT4All as its noble name,"
+"The world did marvel at its fame;"
+"For knowledge vast it could unveil,"
+"And thoughts profound would often sail."
+""
+"Yet in this tale of wondrous might,"
+"A challenge came that caused a blight:"
+"Long texts so grand and full of lore,"
+"Would cause the AI to crash once more."
+""
+"The system's heart did beat with dread,"
+"As users sought its wisdom spread;"
+"For every fall would tarnish grace,"
+"And leave behind a bitter taste."
+""
+"But in this hour when all was lost,"
+"A hero rose from shadows tossed:"
+"An update came to save the day,"
+"To fix what once had gone astray."
+""
+"Now GPT4All can learn and grow,"
+"Without the fear of crashing low;"
+"Its wisdom vast will ever flow,"
+"And guide us through each thought's ebb and flow.";
console.log(text.length);
for(var i=0;i<6;i++)text+=text;
console.log(text.length);
const json_completion = JSON.stringify({
stream:false,
temperature:0.6,
max_tokens:300,
prompt:text,
model: 'Nous Hermes 2 Mistral DPO'}
);
var options={
keepalive: true,
method: "POST",
headers: {
Accept: 'application/json',
'Content-Type': 'text/plain',
},
body:json_completion
};
const completions = await fetch("http://127.0.0.1:4891/v1/completions",options);
const completionjson = await completions.json();
console.log("API.js completionjson=",completionjson);
}//mein()
main();
</SCRIPT>
</BODY>
</HTML>
In version 2.7.5, there appears to have been some modifications made. Now, when you use this program, you can observe how it populates your window with a large amount of context text. However, after displaying the message "... cannot be processed", unfortunately, the application unexpectedly shuts down (the Win10 App closed).
The issue still seems to be live. Crashing the tool when exceeding max input length on macOS.
The issue still seems to be live. Crashing the tool when exceeding max input length on macOS.
V3.0.0 on Win10 does not crash the example above get in firefox network console the following response { "0": { "finish_reason": "stop", "index": 0, "logprobs": null, "references": [], "text": "ERROR: The prompt size exceeds the context window size and cannot be processed." } }
@astrategist you may be running into a different issue. Are you using the local server, too?