bug: Can't run on Apple M1 Pro

[iwootten]

Describe the bug

Unable to run a model on my M1 Pro 16GB. Tried both Mistral and TinyLlama on both 0.4.3 and the nightly version. Original Issue here, updated to Sonama and same thing: https://github.com/janhq/jan/issues/1503

Steps to reproduce Steps to reproduce the behavior:

Download a model Create a thread using said model Enter a prompt See error

Expected behavior The model gives some sort of response.

Screenshots

Environment details

• Operating System: MacOS Sonoma 14.2.1
• Jan Version: 0.4.3/0.4.3-139
• Processor Apple M1 Pro
• RAM: 16GB

[Foul-Tarnished]

Does Jan even use Metal backend to use Apple GPU ? Readme just says Apple is supported and is fast.

Also we can't switch between cpu and gpu (metal) in the settings, like we can for nvidia gpu users

[freelerobot]

@Van-QA can we follow up here when you are back tomorrow and get a full diagnosis?

Side note: I have the exact same specs and am able to run things.

In the meantime, can I trouble you to do a clean install: https://jan.ai/troubleshooting/stuck-on-broken-build/

So sorry for this

[iwootten]

@0xSage Did you mean to include Van, since it was me who raised this?

This issue was raised after a clean install...just been through all the removal steps above on the broken build and same thing. The application cache folder says it doesn't exist on removal.

On the plus side I do get fancy animated wallpapers in Sonama now so it's not all bad.

[louis-jan]

@iwootten sorry for asking, but could you please help try nightly build which would generate ~jan/logs. The log files would help us to investigating this issue.

[iwootten]

Sure, like I said I already ran nightly - here's the logfile it generated:

https://gist.github.com/iwootten/6cce37433abaaad3f3df039c50a496e2

Looks like it's failing to load the model.

[louis-jan]

cc @tikikun

[NITRO]::Error: .......GGML_ASSERT: /nitro/llama.cpp/ggml-backend.c:1270: (char *)addr + ggml_backend_buffer_get_alloc_size(buffer, tensor) <= (char *)ggml_backend_buffer_get_base(buffer) + ggml_backend_buffer_get_size(buffer)

[tikikun]

https://github.com/ggerganov/llama.cpp/issues/2708

@hahuyhoang411 can we have any way around this?

[louis-jan]

Sure, like I said I already ran nightly - here's the logfile it generated:

https://gist.github.com/iwootten/6cce37433abaaad3f3df039c50a496e2

Looks like it's failing to load the model.

This seems like an OOM issue. Could you please try using the latest nightly build and share the result?

Also, there are several version bumps that may have already addressed the first issue.

[louis-jan]

I'm sorry to ask, but could you please try again with the latest release? We have recently fixed some issues and updated the core engine.