bug: model breaks and repeats itself after long use

[TempUser13]

When I'm using Mistral Instruct 7B Q4, after a lot of outputted text the model will just break repeating itself. I'm not sure if this is a Model issue or Jan.

Steps to reproduce the behavior: Use the model like usual and eventually after its outputted lots of paragraphs it'll break.

• Operating System: MacOS Sonoma 14.3.1 on a M1 Pro MacBook Pro
• Jan Version: 0.4.9
• Processor: M1 Pro (8P/2E), 16 GPU Cores
• RAM: 16GB

Logs jan.txt

[Propheticus]

Could very well be a limitation of the model. For the same reason online bots are often limited to a max of 20 messages per conversation. You could try increasing the context (ctx_len) in model.json ... the v0.2 version of Mistral can go up to a context of 32k while Jan's UI normally limits to 4096, but beware that takes more (v)RAM and performs slower.

[SmokeShine]

facing a similar issue. "Difference between Mistral-T and transformer" starts repeating after 4 points.

[hahuyhoang411]

Thanks @Propheticus for the answer. This is related to the maximum text the AI can handle. We will have a better notification to point out the maximum cap.

cc @SmokeShine @TempUser13

[TempUser13]

Could very well be a limitation of the model. For the same reason online bots are often limited to a max of 20 messages per conversation. You could try increasing the context (ctx_len) in model.json ... the v0.2 version of Mistral can go up to a context of 32k while Jan's UI normally limits to 4096, but beware that takes more (v)RAM and performs slower.

This fixed my issue and I forgot to reply, thank you.

Thanks @Propheticus for the answer. This is related to the maximum text the AI can handle. We will have a better notification to point out the maximum cap.

cc @SmokeShine @TempUser13

Thank you.