jan icon indicating copy to clipboard operation
jan copied to clipboard

Published 20 hours ago • verified publisher icon janhq

bug: Model large than 10 gb don't work correctly

Open raveismadly opened this issue 1 year ago • 3 comments

Describe the bug In attached video i run model Phind 34B to show bug

Steps to reproduce Steps to reproduce the behavior:

  1. Download any recommended model large than 10 gb and small than 30 gb
  2. Click on 'Regenerate or start new thread'
  3. The model is not working correctly. When launched, the speed is very slow. Moreover, either it writes an empty line or incoherent words. Regardless of the model, what matters most is that it is over 10 GB and less than 30 GB

Expected behavior Run model in recommended list and it work correctly.

Environment details

  • Operating System: [MacOS Sonoma 14.2.1]
  • Jan Version: [0.4.5]
  • Processor: [Apple M1]
  • RAM: [32GB]

Additional context Add any other context or information that could be helpful in diagnosing the problem.

https://github.com/janhq/jan/assets/36083130/58fa81df-8c3e-487c-8c3e-6f2ef18b38f3

raveismadly avatar Jan 30 '24 17:01 raveismadly

Please help us by providing the app.log for troubleshooting: https://jan.ai/troubleshooting/how-to-get-error-logs/

Van-QA avatar Jan 31 '24 04:01 Van-QA

app.log My app log

raveismadly avatar Jan 31 '24 06:01 raveismadly

Hi there,

maybe the problem has to do with not enough memory

I tested with the model phind-codellama-34b-v2.Q5_K_M.gguf on my MacBook Pro 64GB RAM and it works without errors.

Here is my configuration.

Environment details

  • MacBook Pro
  • Operating System: [MacOS Sonoma 14.3]
  • Jan Version: [0.4.5]
  • Processor: [Apple M1 Max]
  • RAM: [64GB]

Additional context I have attached my app.log (phind34b_app.log) for comparison. Hope it will help you with the analysis.

In @raveismadly 's app.log you can see repeated NITRO errors after update_slots, e.g. B. after lines 4553, 5067, 5283, 5309 etc.

[NITRO]::Error: ggml_metal_graph_compute: command buffer 5 failed with status 5

You can't see these errors in my log file.

Kind regards

Matthias

phind34b_app.log

m-kroeger avatar Feb 02 '24 14:02 m-kroeger

Hi @Van-QA What's the cause of this?

imtuyethan avatar Feb 07 '24 02:02 imtuyethan

hi, we will investigate this issue as an OOM case, thank you.

Hi @Van-QA What's the cause of this?

Van-QA avatar Feb 07 '24 02:02 Van-QA

Seems like the problem is because you run a model that is too big for your RAM. At this point it's our fault that don't deliver a clear error message. This issue is addressed in this ticket, you can track the progress here: https://github.com/janhq/jan/issues/1627

imtuyethan avatar Feb 07 '24 02:02 imtuyethan