LocalAI icon indicating copy to clipboard operation
LocalAI copied to clipboard

Published 20 hours ago • verified publisher icon mudler

LocalAI sends empty chunk to chatbot_ui and closes stream

Open ga-it opened this issue 1 year ago • 5 comments

LocalAI version: LocalAI version: v2.10.1-23-gbd25d80

Environment, CPU architecture, OS, and Version: Xen VM on Debian api_1 | CPU info: api_1 | model name : Intel(R) Xeon(R) CPU E5-4620 0 @ 2.20GHz api_1 | flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc rep_good nopl cpuid tsc_known_freq pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx hypervisor lahf_lm xsaveopt arch_capabilities api_1 | CPU: AVX found OK api_1 | CPU: no AVX2 found api_1 | CPU: no AVX512 found

Describe the bug The issue seems to be that localai sends an empty chunk to Chatbot_ui which closes the connection to further streaming:

api_1 | 12:34PM DBG Sending chunk: {"created":1711281224,"object":"chat.completion.chunk","id":"XXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"role":"assistant","content":""}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

Visually, you can see the Chatbot UI close the waiting prompt.

Then in the debug logs you can see the following chunks streamed - but these do not reach the chatbot_ui interface.

To Reproduce

Install with configuration below. Send input from Chatbot_ui interface. Bot cursor flashes and then goes "silent."

Expected behavior

Input to bot waits for streamed response from localai and accepts and reproduces streamed output until complete.

Logs Here is the initial request in the logs:

api_1 | 12:51PM DBG Request received: {"model":"gpt-3.5-turbo","language":"","n":0,"top_p":null,"top_k":null,"temperature":0.5,"max_tokens":1000,"echo":false,"batch":0,"ignore_eos":false,"repeat_penalty":0,"n_keep":0,"frequency_penalty":0,"presence_penalty":0,"tfz":0,"typical_p":0,"seed":null,"negative_prompt":"","rope_freq_base":0,"rope_freq_scale":0,"negative_prompt_scale":0,"use_fast_tokenizer":false,"clip_skip":0,"tokenizer":"","file":"","response_format":{},"size":"","prompt":null,"instruction":"","input":null,"stop":null,"messages":[{"role":"system","content":"You are ChatGPT, a large language model trained by OpenAI. Follow the user's instructions carefully. Respond using markdown."},{"role":"user","content":"hello"},{"role":"assistant","content":""},{"role":"user","content":"help"},{"role":"assistant","content":""},{"role":"user","content":"hello"},{"role":"assistant","content":""},{"role":"user","content":"hello"},{"role":"assistant","content":""},{"role":"user","content":"hello"}],"functions":null,"function_call":null,"stream":true,"mode":0,"step":0,"grammar":"","grammar_json_functions":null,"backend":"","model_base_name":""} api_1 | 12:51PM DBG Configuration read: &{PredictionOptions:{Model:ggml-gpt4all-j.bin Language: N:0 TopP:0xc000bc8190 TopK:0xc000bc8188 Temperature:0xc000816498 Maxtokens:0xc000816478 Echo:false Batch:0 IgnoreEOS:false RepeatPenalty:0 Keep:0 FrequencyPenalty:0 PresencePenalty:0 TFZ:0 TypicalP:0 Seed:0xc000bc8218 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:gpt-3.5-turbo F16:0xc000bc81c8 Threads:0xc000bc81c0 Debug:0xc0008166b8 Roles:map[] Embeddings:false Backend:gpt4all-j TemplateConfig:{Chat:gpt4all-chat ChatMessage: Completion:gpt4all-completion Edit: Functions:} PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: FunctionsConfig:{DisableNoAction:false NoActionFunctionName: NoActionDescriptionName: ParallelCalls:false} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0xc000bc8200 MirostatTAU:0xc000bc81f8 Mirostat:0xc000bc81f0 NGPULayers:0xc000bc8208 MMap:0xc000bc8210 MMlock:0xc000bc8211 LowVRAM:0xc000bc8211 Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] TrimSuffix:[] ContextSize:0xc000bc8148 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 MMProj: RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{CUDA:false PipelineType: SchedulerType: EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} VallE:{AudioPath:} CUDA:false DownloadFiles:[] Description: Usage:} api_1 | 12:51PM DBG Parameters: &{PredictionOptions:{Model:ggml-gpt4all-j.bin Language: N:0 TopP:0xc000bc8190 TopK:0xc000bc8188 Temperature:0xc000816498 Maxtokens:0xc000816478 Echo:false Batch:0 IgnoreEOS:false RepeatPenalty:0 Keep:0 FrequencyPenalty:0 PresencePenalty:0 TFZ:0 TypicalP:0 Seed:0xc000bc8218 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:gpt-3.5-turbo F16:0xc000bc81c8 Threads:0xc000bc81c0 Debug:0xc0008166b8 Roles:map[] Embeddings:false Backend:gpt4all-j TemplateConfig:{Chat:gpt4all-chat ChatMessage: Completion:gpt4all-completion Edit: Functions:} PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: FunctionsConfig:{DisableNoAction:false NoActionFunctionName: NoActionDescriptionName: ParallelCalls:false} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0xc000bc8200 MirostatTAU:0xc000bc81f8 Mirostat:0xc000bc81f0 NGPULayers:0xc000bc8208 MMap:0xc000bc8210 MMlock:0xc000bc8211 LowVRAM:0xc000bc8211 Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] TrimSuffix:[] ContextSize:0xc000bc8148 NUMA:false LoraAdapter: LoraBase: LoraScale:0 NoMulMatQ:false DraftModel: NDraft:0 Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 MMProj: RopeScaling: ModelType: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{CUDA:false PipelineType: SchedulerType: EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder: ControlNet:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} VallE:{AudioPath:} CUDA:false DownloadFiles:[] Description: Usage:} api_1 | 12:51PM DBG Prompt (before templating): You are ChatGPT, a large language model trained by OpenAI. Follow the user's instructions carefully. Respond using markdown. api_1 | hello api_1 | api_1 | help api_1 | api_1 | hello api_1 | api_1 | hello api_1 | api_1 | hello api_1 | 12:51PM DBG Stream request received api_1 | 12:51PM DBG Template found, input modified to: The prompt below is a question to answer, a task to complete, or a conversation to respond to; decide which and write an appropriate response. api_1 | ### Prompt: api_1 | You are ChatGPT, a large language model trained by OpenAI. Follow the user's instructions carefully. Respond using markdown. api_1 | hello api_1 | api_1 | help api_1 | api_1 | hello api_1 | api_1 | hello api_1 | api_1 | hello api_1 | ### Response: api_1 | api_1 | 12:51PM DBG Prompt (after templating): The prompt below is a question to answer, a task to complete, or a conversation to respond to; decide which and write an appropriate response. api_1 | ### Prompt: api_1 | You are ChatGPT, a large language model trained by OpenAI. Follow the user's instructions carefully. Respond using markdown. api_1 | hello api_1 | api_1 | help api_1 | api_1 | hello api_1 | api_1 | hello api_1 | api_1 | hello api_1 | ### Response: api_1 | api_1 | [172.25.0.3]:54420 200 - POST /v1/chat/completions api_1 | 12:51PM INF Loading model 'ggml-gpt4all-j.bin' with backend gpt4all-j api_1 | 12:51PM DBG Loading model in memory from file: /models/ggml-gpt4all-j.bin api_1 | 12:51PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"xxxx","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"role":"assistant","content":""}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

api_1 | api_1 | 12:51PM DBG Loading Model ggml-gpt4all-j.bin with gRPC (file: /models/ggml-gpt4all-j.bin) (backend: gpt4all): {backendString:gpt4all-j model:ggml-gpt4all-j.bin threads:44 assetDir:/tmp/localai/backend_data context:{emptyCtx:{}} gRPCOptions:0xc000140200 externalBackends:map[my-backend:127.0.0.1:9000 my-backend2:/usr/bin/backend.py] grpcAttempts:20 grpcAttemptsDelay:2 singleActiveBackend:false parallelRequests:true} api_1 | 12:51PM DBG Loading GRPC Process: /tmp/localai/backend_data/backend-assets/grpc/gpt4all api_1 | 12:51PM DBG GRPC Service for ggml-gpt4all-j.bin will be running at: '127.0.0.1:46537' api_1 | 12:51PM DBG GRPC Service state dir: /tmp/go-processmanager3274546950 api_1 | 12:51PM DBG GRPC Service Started api_1 | 12:51PM DBG GRPC(ggml-gpt4all-j.bin-127.0.0.1:46537): stderr 2024/03/24 12:51:26 gRPC Server listening at 127.0.0.1:46537 api_1 | 12:51PM DBG GRPC Service Ready api_1 | 12:51PM DBG GRPC: Loading model with options: {state:{NoUnkeyedLiterals:{} DoNotCompare:[] DoNotCopy:[] atomicMessageInfo:} sizeCache:0 unknownFields:[] Model:ggml-gpt4all-j.bin ContextSize:1024 Seed:1113555745 NBatch:512 F16Memory:false MLock:false MMap:true VocabOnly:false LowVRAM:false Embeddings:false NUMA:false NGPULayers:99999999 MainGPU: TensorSplit: Threads:44 LibrarySearchPath:/tmp/localai/backend_data/backend-assets/gpt4all RopeFreqBase:0 RopeFreqScale:0 RMSNormEps:0 NGQA:0 ModelFile:/models/ggml-gpt4all-j.bin Device: UseTriton:false ModelBaseName: UseFastTokenizer:false PipelineType: SchedulerType: CUDA:false CFGScale:0 IMG2IMG:false CLIPModel: CLIPSubfolder: CLIPSkip:0 ControlNet: Tokenizer: LoraBase: LoraAdapter: LoraScale:0 NoMulMatQ:false DraftModel: AudioPath: Quantization: GPUMemoryUtilization:0 TrustRemoteCode:false EnforceEager:false SwapSpace:0 MaxModelLen:0 MMProj: RopeScaling: YarnExtFactor:0 YarnAttnFactor:0 YarnBetaFast:0 YarnBetaSlow:0 Type:} api_1 | 12:51PM DBG GRPC(ggml-gpt4all-j.bin-127.0.0.1:46537): stdout gptj_model_load: loading model from '/models/ggml-gpt4all-j.bin' - please wait ... api_1 | 12:51PM DBG GRPC(ggml-gpt4all-j.bin-127.0.0.1:46537): stdout gptj_model_load: n_vocab = 50400 api_1 | 12:51PM DBG GRPC(ggml-gpt4all-j.bin-127.0.0.1:46537): stdout gptj_model_load: n_ctx = 2048 api_1 | 12:51PM DBG GRPC(ggml-gpt4all-j.bin-127.0.0.1:46537): stdout gptj_model_load: n_embd = 4096 api_1 | 12:51PM DBG GRPC(ggml-gpt4all-j.bin-127.0.0.1:46537): stdout gptj_model_load: n_head = 16 api_1 | 12:51PM DBG GRPC(ggml-gpt4all-j.bin-127.0.0.1:46537): stdout gptj_model_load: n_layer = 28 api_1 | 12:51PM DBG GRPC(ggml-gpt4all-j.bin-127.0.0.1:46537): stdout gptj_model_load: n_rot = 64 api_1 | 12:51PM DBG GRPC(ggml-gpt4all-j.bin-127.0.0.1:46537): stdout gptj_model_load: f16 = 2 api_1 | 12:51PM DBG GRPC(ggml-gpt4all-j.bin-127.0.0.1:46537): stdout gptj_model_load: ggml ctx size = 5401.45 MB api_1 | 12:51PM DBG GRPC(ggml-gpt4all-j.bin-127.0.0.1:46537): stdout gptj_model_load: kv self size = 896.00 MB api_1 | 12:51PM DBG GRPC(ggml-gpt4all-j.bin-127.0.0.1:46537): stdout gptj_model_load: ................................... done api_1 | 12:51PM DBG GRPC(ggml-gpt4all-j.bin-127.0.0.1:46537): stdout gptj_model_load: model size = 3609.38 MB / num tensors = 285 api_1 | 12:51PM DBG [WatchDog] Watchdog checks for idle connections api_1 | [127.0.0.1]:39914 200 - GET /readyz api_1 | 12:52PM DBG [WatchDog] Watchdog checks for idle connections api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"H"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"e"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"l"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"l"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"o"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":","}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"I"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"a"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"m"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"C"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"h"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"a"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"t"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"G"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"P"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"T"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"a"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"n"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"d"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"w"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"o"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"u"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"l"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"d"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"b"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"e"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"h"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"a"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"p"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"p"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"y"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"t"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"o"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"a"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"s"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"s"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"i"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"s"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"t"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"y"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"o"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"u"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"."}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"P"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"l"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"e"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"a"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"s"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"e"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"p"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"r"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"o"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"v"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"i"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"d"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"e"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"m"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"e"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"w"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"i"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"t"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"h"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"m"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"o"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"r"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"e"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"s"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"p"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"e"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"c"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"i"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"f"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"i"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"c"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"i"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"n"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"s"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"t"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"r"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"u"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"c"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"t"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"i"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"o"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"n"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"s"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"o"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"r"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"a"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"t"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"a"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"s"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"k"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"t"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"o"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"c"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"o"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"m"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"p"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"l"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"e"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"t"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"e"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG Sending chunk: {"created":1711284644,"object":"chat.completion.chunk","id":"XXXXXXXXXXX","model":"gpt-3.5-turbo","choices":[{"index":0,"finish_reason":"","delta":{"content":"."}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} api_1 | api_1 | 12:52PM DBG [WatchDog] Watchdog checks for idle connections api_1 | 12:52PM DBG [WatchDog] 127.0.0.1:46537: idle connection api_1 | [127.0.0.1]:57322 200 - GET /readyz api_1 | 12:53PM DBG [WatchDog] Watchdog checks for idle connections api_1 | 12:53PM DBG [WatchDog] 127.0.0.1:46537: idle connection api_1 | 12:53PM DBG [WatchDog] Watchdog checks for idle connections api_1 | 12:53PM DBG [WatchDog] 127.0.0.1:46537: idle connection api_1 | [127.0.0.1]:59102 200 - GET /readyz api_1 | 12:54PM DBG [WatchDog] Watchdog checks for idle connections api_1 | 12:54PM DBG [WatchDog] 127.0.0.1:46537: idle connection

Additional context Dom0 Xen info:

release : 6.6.15-amd64 version : #1 SMP PREEMPT_DYNAMIC Debian 6.6.15-2 (2024-02-04) machine : x86_64 nr_cpus : 64 max_cpu_id : 63 nr_nodes : 4 cores_per_socket : 8 threads_per_core : 2 cpu_mhz : 2200.000 hw_caps : bfebfbff:17bee3ff:2c100800:00000001:00000001:00000000:00000000:00000100 virt_caps : pv hvm hvm_directio pv_directio hap shadow gnttab-v1 gnttab-v2 total_memory : 262098 free_memory : 158655 sharing_freed_memory : 0 sharing_used_memory : 0 outstanding_claims : 0 free_cpus : 0 xen_major : 4 xen_minor : 17 xen_extra : .4-pre xen_version : 4.17.4-pre xen_caps : xen-3.0-x86_64 hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit2 xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : xen_commandline : placeholder dom0_mem=2G,max:2G dom0_max_vcpus=4 dom0_vcpus_pin noxsaveopt cc_compiler : x86_64-linux-gnu-gcc (Debian 13.2.0-13) 13.2.0 cc_compile_by : pkg-xen-devel cc_compile_domain : lists.alioth.debian.org cc_compile_date : Sun Feb 4 12:45:17 UTC 2024 build_id : 83abdcea4bee1c3a4532d44649e2caec19737228 xend_config_format : 4

xl list Name ID Mem VCPUs State Time(s) Domain-0 0 2048 4 r----- 1325.6 XXX 1 32768 8 -b---- 9270.8 localai 5 65536 52 -b---- 14042.3

Localai xen domu config:

kernel = '/usr/lib/grub-xen/grub-x86_64-xen.bin' extra = '(hd1)/boot/grub/grub.cfg'

xen_platform_pci=1 pae=1 acpi=1 apic=1

vcpus = '52' memory = '65536' usb = 1

device_model_version = 'qemu-xen'

disk = ['/dev/XXX-disk,,xvda,w','/dev/XXX-swap,,xvdb,w','/dev/sdb2,,xvdc,w']

name = 'XXXX' vif = ['bridge=xenbr1']

on_poweroff = 'destroy' on_reboot = 'restart' on_crash = 'restart' on_xend_stop = 'shutdown' on_xend_start = 'start'

LocalAI/.env THREADS=44

CONTEXT_SIZE=256 GALLERIES=[{"name":"model-gallery", "url":"github:go-skynet/model-gallery/index.yaml"}]

CORS=true CORS_ALLOW_ORIGINS=XXX

MODELS_PATH=/models

BUILD_TYPE=openblas

GO_TAGS=stablediffusion tts

IMAGE_PATH=/tmp

EXTERNAL_GRPC_BACKENDS=my-backend:127.0.0.1:9000,my-backend2:/usr/bin/backend.py

HUGGINGFACE_HUB_CACHE=/usr/local/huggingface

LLAMACPP_PARALLEL=2

PARALLEL_REQUESTS=true

WATCHDOG_IDLE=true WATCHDOG_IDLE_TIMEOUT=5m

LocalAI/docker-compose.yaml * version: '3.6'

services: api: image: quay.io/go-skynet/local-ai:master-ffmpeg build: context: . dockerfile: Dockerfile args: - IMAGE_TYPE=extras - BASE_IMAGE=ubuntu:22.04 ports: - 8080:8080

env_file:
  - .env
environment:
  - MODELS_PATH=/models
  - DEBUG=true
  - 'PRELOAD_MODELS=[{"url": "github:go-skynet/model-gallery/gpt4all-j.yaml", "name": "gpt-3.5-turbo"}]'
volumes:
  - ./models:/models:cached
  - ./images/:/tmp/generated/images/
command:
- phi-2
cpus: '48.0'
mem_limit: 40G

chatgpt: depends_on: api: condition: service_healthy image: ghcr.io/mckaywrigley/chatbot-ui:main ports: - 3000:3000 environment: - 'OPENAI_API_KEY=sk-XXXXXXXXXXXXXXXXXXXX' - 'OPENAI_API_HOST=http://api:8080' cpus: '1.0' mem_limit: 2G

ga-it avatar Mar 24 '24 14:03 ga-it

same problem here.

gillbates avatar Mar 25 '24 05:03 gillbates

Hi @mudler

I appreciate all your great work and workload

Any word on the above? Is it my misconfiguration or is this a bone fide bug?

I am stuck without a resolution path.

Regards

ga-it avatar Mar 26 '24 17:03 ga-it

I usually wouldn't add anything but because of the label "unconfirmed" I wanted to say "me too". I haven't been able to find the rootcause, a same version works but all of a sudden doesn't anymore. I might have updated my system inbetween, which could explain that. I use my GPU nvidia with the https://github.com/Robitx/gp.nvim plugin. It fails all the time now, even on new sessions.

... $ nix run .#local-ai-cublas -- --models-path ~/localai-models --autoload-galleries --address ":11111" --debug .... <|im_start|>assistant

[127.0.0.1]:51000 200 - POST /v1/chat/completions 1:23AM DBG Sending chunk: {"created":1711585346,"object":"chat.completion.chunk","id":"868f2609-0af6-4e96-9e92-ff3d7fc84aca","model":"mistral","choices":[{"index":0,"finish_reason":"","delta":{"role":"assistant","content":""}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}} This

'data: {"created":1711585346,"object":"chat.completion.chunk","id":"868f2609-0af6-4e96-9e92-ff3d7fc84aca","model":"mistral","choices":[{"index":0,"finish_reason":"","delta":{"role":"assistant","content":""}}],"usage":{"prompt_tokens":0,"completion_tokens":
0,"total_tokens":0}}\ndata: {"created":1711585346,"object":"chat.completion.chunk","id":"868f2609-0af6-4e96-9e92-ff3d7fc84aca","model":"mistral","choices":[{"index":0,"finish_reason":"stop","delta":{"content":""}}],"usage":{"prompt_tokens":0,"completion_to
kens":0,"total_tokens":0}}\ndata: [DONE]\n'

and as I was writing this message, I realized I started adding recently the --autoload-galleries and without it localAI now works again \o/ I am not sure what the flag does but looks like a tricky one !

teto avatar Mar 28 '24 00:03 teto

Same issue here. Im able to send 1-2 messages and get responses back then it just stops.

Logs

`2024-04-05 20:21:19 6:21PM DBG Model already loaded in memory: 5c7cd056ecf9a4bb5b527410b97f48cb
2024-04-05 20:21:19 6:21PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"role":"assistant","content":""}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:21:19 
2024-04-05 20:21:19 6:21PM DBG Model '5c7cd056ecf9a4bb5b527410b97f48cb' already loaded
2024-04-05 20:21:19 6:21PM DBG GRPC(5c7cd056ecf9a4bb5b527410b97f48cb-127.0.0.1:43219): stdout {"timestamp":1712341279,"level":"INFO","function":"launch_slot_with_data","line":884,"message":"slot is processing task","slot_id":0,"task_id":58}
2024-04-05 20:21:19 6:21PM DBG GRPC(5c7cd056ecf9a4bb5b527410b97f48cb-127.0.0.1:43219): stdout {"timestamp":1712341279,"level":"INFO","function":"update_slots","line":1783,"message":"kv cache rm [p0, end)","slot_id":0,"task_id":58,"p0":0}
2024-04-05 20:21:23 6:21PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"\n"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:21:23 
2024-04-05 20:21:23 6:21PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"\n"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:21:23 
2024-04-05 20:21:23 6:21PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"U"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:21:23 
2024-04-05 20:21:23 6:21PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"n"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:21:23 
2024-04-05 20:21:23 6:21PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"d"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:21:23 
2024-04-05 20:21:23 6:21PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"e"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:21:23 
2024-04-05 20:21:23 6:21PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"r"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:21:23 
2024-04-05 20:21:23 6:21PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":" "}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:21:23 
2024-04-05 20:21:23 6:21PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"k"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:21:23 
2024-04-05 20:21:23 6:21PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"o"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:21:23 
2024-04-05 20:21:23 6:21PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"m"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:21:23 
2024-04-05 20:21:23 6:21PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"m"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:21:23 
2024-04-05 20:21:23 6:21PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"u"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:21:23 
2024-04-05 20:21:23 6:21PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"n"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:21:23 
2024-04-05 20:21:23 6:21PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"f"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:21:23 
2024-04-05 20:21:23 6:21PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"u"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:21:23 
2024-04-05 20:21:23 6:21PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"l"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:21:23 
2024-04-05 20:21:23 6:21PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"l"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:21:23 
2024-04-05 20:21:23 6:21PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"m"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:21:23 
2024-04-05 20:21:23 6:21PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"ä"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

Keeps going like this untill it stops

2024-04-05 20:23:53 6:23PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"\n"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

2024-04-05 20:23:53 
2024-04-05 20:23:53 6:23PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"\n"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:23:53 
2024-04-05 20:23:53 6:23PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"\n"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:23:53 
2024-04-05 20:23:54 6:23PM DBG Sending chunk: {"created":1712341017,"object":"chat.completion.chunk","id":"d28dfe6e-75ec-4fea-b74a-a69f6e2afafd","model":"gpt-4","choices":[{"index":0,"finish_reason":"","delta":{"content":"\n"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
2024-04-05 20:23:54 
2024-04-05 20:23:54 6:23PM DBG GRPC(5c7cd056ecf9a4bb5b527410b97f48cb-127.0.0.1:43219): stdout {"timestamp":1712341434,"level":"INFO","function":"print_timings","line":327,"message":"prompt eval time     =    3762.01 ms /  1559 tokens (    2.41 ms per token,   414.41 tokens per second)","slot_id":0,"task_id":58,"t_prompt_processing":3762.013,"num_prompt_tokens_processed":1559,"t_token":2.413093649775497,"n_tokens_second":414.4057981724146}
2024-04-05 20:23:54 6:23PM DBG GRPC(5c7cd056ecf9a4bb5b527410b97f48cb-127.0.0.1:43219): stdout {"timestamp":1712341434,"level":"INFO","function":"print_timings","line":341,"message":"generation eval time =  150698.70 ms /  2048 runs   (   73.58 ms per token,    13.59 tokens per second)","slot_id":0,"task_id":58,"t_token_generation":150698.697,"n_decoded":2048,"t_token":73.58334814453124,"n_tokens_second":13.59003123961981}
2024-04-05 20:23:54 6:23PM DBG GRPC(5c7cd056ecf9a4bb5b527410b97f48cb-127.0.0.1:43219): stdout {"timestamp":1712341434,"level":"INFO","function":"print_timings","line":351,"message":"          total time =  154460.71 ms","slot_id":0,"task_id":58,"t_prompt_processing":3762.013,"t_token_generation":150698.697,"t_total":154460.71}
2024-04-05 20:23:54 6:23PM DBG GRPC(5c7cd056ecf9a4bb5b527410b97f48cb-127.0.0.1:43219): stdout {"timestamp":1712341434,"level":"INFO","function":"update_slots","line":1594,"message":"slot released","slot_id":0,"task_id":58,"n_ctx":4096,"n_past":3606,"n_system_tokens":0,"n_cache_tokens":3607,"truncated":false}

LocalAI version: Docker using docker-compose: Image version: 7e498578e3fd

version: "3.9"
services:
  api:
    image: localai/localai:latest-aio-gpu-nvidia-cuda-12
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/readyz"]
      interval: 1m
      timeout: 20m
      retries: 5
    ports:
      - 8080:8080
    environment:
      - DEBUG=true
      # ...
    volumes:
      - ./models:/build/models:cached
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Environment, CPU architecture, OS, and Version: WSL2- Ubuntu 22.04 Linux GIBBSTATION 5.15.146.1-microsoft-standard-WSL2 #1 SMP Thu Jan 11 04:09:03 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

CPU info: 2024-04-05 20:33:03 model name : AMD Ryzen 5 5600X 6-Core Processor 2024-04-05 20:33:03 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr arat npt nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload umip vaes vpclmulqdq rdpid fsrm

NVIDIA GPU detected via WSL2 2024-04-05 20:33:03 Fri Apr 5 18:33:03 2024
2024-04-05 20:33:03 +---------------------------------------------------------------------------------------+ 2024-04-05 20:33:03 | NVIDIA-SMI 545.23.06 Driver Version: 545.92 CUDA Version: 12.3 | 2024-04-05 20:33:03 |-----------------------------------------+----------------------+----------------------+ 2024-04-05 20:33:03 | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-04-05 20:33:03 | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-04-05 20:33:03 | | | MIG M. | 2024-04-05 20:33:03 |=========================================+======================+======================| 2024-04-05 20:33:03 | 0 NVIDIA GeForce RTX 3070 On | 00000000:2B:00.0 On | N/A | 2024-04-05 20:33:03 | 56% 46C P3 47W / 270W | 1189MiB / 8192MiB | 27% Default | 2024-04-05 20:33:03 | | | N/A | 2024-04-05 20:33:03 +-----------------------------------------+----------------------+----------------------+ 2024-04-05 20:33:03
2024-04-05 20:33:03 +---------------------------------------------------------------------------------------+ 2024-04-05 20:33:03 | Processes: | 2024-04-05 20:33:03 | GPU GI CI PID Type Process name GPU Memory | 2024-04-05 20:33:03 | ID ID Usage | 2024-04-05 20:33:03 |=======================================================================================| 2024-04-05 20:33:03 | 0 N/A N/A 31 G /Xwayland N/A | 2024-04-05 20:33:03 | 0 N/A N/A 33 G /Xwayland N/A | 2024-04-05 20:33:03 | 0 N/A N/A 35 G /Xwayland N/A | 2024-04-05 20:33:03 +---------------------------------------------------------------------------------------+ 2024-04-05 20:33:03 NVIDIA GPU detected. Attempting to find memory size... 2024-04-05 20:33:03 Total GPU Memory: 8192 MiB

EDIT After doing some troubleshooting it seems that the issue only occurs when trying to have a conversation around embedded documents. When curling the API with sentences manually im able to have it working without it throwing any errors.

s0undy avatar Apr 05 '24 18:04 s0undy

It seems like the issue you're experiencing is related to sending multiple messages in a conversation involving embedded documents. This issue might be due to a bug in one of the recent updates of the LocalAI version you are using. However, I have found a workaround for you to continue using the chat functionality while this issue is being investigated and fixed.

Workaround: When sending multiple messages in a conversation involving embedded documents, split the messages into separate chat completion requests with a delay between them. For example, instead of sending:

{
  "object": "chat.completion.chunk",
  "choices": [
  {
    "index": 0,
    "finish_reason": "",
    "delta": {
      "content": 'Message 1...',
      "children": [
        {
          "object

localai-bot avatar Apr 15 '24 17:04 localai-bot