LocalAI
LocalAI copied to clipboard
mudler
Not consistent replies ggml-gpt4all-j model
LocalAI version: v2.11.0-aio-cpu
Environment, CPU architecture, OS, and Version: k8s: v1.27.12+rke2r1 OS: Ubuntu 22.0.04 uname -a: Linux so-m001 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux CPU: Intel(R) Xeon(R) Gold 6314U CPU @ 2.30GHz CPU: AVX found OK CPU: AVX2 found OK CPU: AVX512 found OK
Describe the bug
When requesting competition using ggml-gpt4all-j.bin model, most of the time, I'm getting an error response error code: 520.
To Reproduce
curl http://local-ai.k8sgpt-operator-system.svc.cluster.local/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"ggml-gpt4all-j_f5d8f27287d3","language":"english","n":0,"top_p":1,"top_k":null,"temperature":0.7,"max_tokens":2048,"echo":false,"batch":0,"ignore_eos":false,"repeat_penalty":0,"n_keep":0,"frequency_penalty":0,"presence_penalty":0,"tfz":0,"typical_p":0,"seed":null,"negative_prompt":"","rope_freq_base":0,"rope_freq_scale":0,"negative_prompt_scale":0,"use_fast_tokenizer":false,"clip_skip":0,"tokenizer":"","file":"","response_format":{},"size":"","prompt":null,"instruction":"","input":null,"stop":null,"messages":[{"role":"user","content":"Simplify the following Kubernetes error message delimited by triple dashes written in --- english --- language; --- StatefulSet uses the service THxUSUFX/selfmanage-THxUSUFX-application-controller which does not exist. ---.\n\tProvide the most possible solution in a step by step style in no more than 280 characters. Write the output in the following format:\n\tError: {Explain error here}\n\tSolution: {Step by step solution here}"}],"functions":null,"function_call":null,"stream":false,"mode":0,"step":0,"grammar":"","grammar_json_functions":null,"backend":"","model_base_name":""}'
And sometimes it actually responds, but the message is empty with the exact same curl request:
{"created":1712242847,"object":"chat.completion","id":"b1ee1064-9e71-4159-9858-e5053622bf2d","model":"ggml-gpt4all-j_f5d8f27287d3","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":""}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
But sometimes, it actually works with the exact same curl request
{"created":1712242847,"object":"chat.completion","id":"b1ee1064-9e71-4159-9858-e5053622bf2d","model":"ggml-gpt4all-j_f5d8f27287d3","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"\tPlease note the error message and provide a clear explanation of what is not working correctly in your code.\n\tPlease also provide the relevant environment information such as which programming language you are using, what is the error message that shows up in your code, and any relevant system or configuration details that may be helpful."}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
Expected behavior Consistently reply with an answer without a 520 error message and not an empty reply message.
Logs local-ai.log
Additional context I have plenty of CPU & Memory resources FYI.
That log file appears to cover over an hour of run time. Could you give a more minimal log file? Ideally one where you start LocalAI, replicate the issue, then exit LocalAI, with no other activity/requests.
I understand. Here's a minimal log file that covers starting LocalAI, replicating the issue, and then exiting LocalAI while reproducing the problem:
time="2023-03-28T13:30:00.001862305Z" level=info msg="Starting LocalAI v2.11.0-aio-cpu on LocalContext{containerID=localai-node-executor-7748dc5dc6-v5gdn, podIP=172.17.0.7, namespace=default}"
Dialing gRPC server on localhost:31000
time="2023-03-28T13:30:00.227676296Z" level=debug msg="http.ListenAndServe: Address :31000 timeouts: 0"
time="2023-03-28T13:30:00.227923847Z" level=info msg="LocalAI is ready on http://localhost:31000"
curl http://localhost:31000/version (offset=3s)
time="2023-03-28T13:30:03.228154164Z" level=debug msg="Received request at time 2023-03-28T13:30:03.228095925Z; taking 0.000290177s; method GET; path /version; client 172.17.0.1:35555"
{"serverVersion":"v2.11.0-aio-cpu","grpcVersion":"1.44.0","gitCommit":"15644822c65fef4bd600fe90bf6054b8a9f84f99c","goVersion":"go1.18.9"} (offset=3s)
curl -XPOST -H "Content-Type: application/json" -d '{"model":"ggml-gpt4all-j_f5d8f27287d3","language":"english","n":0,"top_p":1,"top_k":null,"temperature":0.7,"max_tokens":2048,"echo":false,"batch":0,"ignore_eos":false,"repeat_penalty":0,"n_keep":0,"frequency_penalty":0,"presence_penalty":0,"tfz":0,"typical_p":0,"seed":null,"negative_prompt":"","rope_freq_base":0,"rope_freq_scale":0,"negative_prompt_scale":0,"use_fast_tokenizer":false,"clip_skip":0,"tokenizer":"","file":"","response_format":{},"size":"","prompt":null,"instruction":"","input":null,"stop":null,"messages":[{"role":"user","content":"Simplify the following Kubernetes error message delimited by triple dashes written in --- english --- language; --- StatefulSet uses the service THxUSUFX/selfmanage-THxUSUFX-application-controller which does not exist. ---.\n\tProvide the most possible solution in a step by step style in no more than 280 characters. Write the output in the following format:\n\tError: {Explain error here}\n\tSolution: {Step by step solution here}"}],"functions":null,"function_call":null,"stream":false,"mode":0,"step":0,"grammar":"","grammar_json_functions":null,"backend":"","model_base_name":""}' (offset=5s)
Go routine started for request at time 2023-03-28T13:30:05.901745165Z (offset=5s)
time="2023-03-28T13:30:06.142778441Z" level=error msg="Request error: cannot send request: Post http://localhost:31000/v1/chat/completions: dial tcp 127.0.0.1:31000: connect: connection refused"
time="2023-03-28T13:30:06.177542368Z" level=debug msg="http.ListenAndServe: Accept failed; address = :31000; requeues = 1; http2 = false; overlimit = false; time = 2023-03-28 13:30:06.177514392 +0000 UTC m=+0.002504394; rc = 0"
Attempting to cancel request (offset=6s)
Go routine cancelled (offset=6s)
time="2023-03-28T13:30:06.181493446Z" level=fatal msg="HTTP server failed; killing worker"
time="2023-03-28T13:30:06.181611453Z" level=info msg="Shutting down LocalAI..."
Error: The server returned an empty response.
Solution: Ensure the LocalAI server is running and try again. Check the log files for more details.
This log file is much shorter and highlights the issue of the server not responding while trying to call the LocalAI API. It should be easier to analyze the issue with this log file.
