bug: Failed! Model Meta-Llama-3.1-8B-Instruct-128k-Q4_0 failed to start.

[SimplyProgrammer]

• [X] I have searched the existing issues

Current behavior

I am unable to start the locally imported model from gpt4all based on Llama-3.1. It simply will not start, sometimes it will display "Failed! Model Meta-Llama-3.1-8B-Instruct-128k-Q4_0 failed to start." in the top corner but even that is not consistent so it is hard to tell what is happening...

Minimum reproduction step

1. Download Meta-Llama-3.1-8B-Instruct-128k-Q4_0.gguf

• In my case, I have downloaded it using gpt4all and then in Jan, I have imported it locally and checked the option to copy it to Jan directory. (I tried both options but neither of them worked...)

2. Try to chat with it -> Nothing happens. Or try to activate it -> it fails...

Expected behavior

The model should start and I should be able to chat with it, or at least get some sort of more consistent error prompts so at least I know that something has happened even if it failed...

Screenshots / Logs

app.log

Jan version

0.5.2

In which operating systems have you tested?

• [ ] macOS
• [X] Windows
• [ ] Linux

Environment details

Windows 10 Intel Core I9 RTX 3080 ti (recently updated from 2080 but it didn't work on it either...) 64GB ram

[Van-QA]

h‌i there, we are working on a huge refa‌c‌tor of Jan, ‌would you mi‌nd trying again with Jan x Cortex via nightly bui‌‌ld? https://github.com/janhq/jan?tab=readme-ov-file#download Thank you

[SimplyProgrammer]

Hi, thanks for the answer. I have tried the nightly build 0.5.2-573 and it indeed works as it should in this version! But unfortunately, this version does not seem to support any kind of hardware (gpu) acceleration and therefore the speed of response generation is, let's say questionable... But I understand that this is nightly build tho, hope it will make it to the full release soon.

[anthonyalayo]

I have the same issue, looking forward to it in the full release.

[anthonyalayo]

For what it's worth, I tried this out on the nightly and it's failing in the same way: https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF

[Van-QA]

Hi there, sorry for the inconvenience, due to a roll back in Jan nightly, we are in progress of making Jan support Llama3.1 again, we will keep you posted in any news 🙏

[louis-jan]

cortex-cpp bump (0.5.0) should address this.

[Van-QA]

tested and looking good uisng Jan v0.5.2-619

[Van-QA]

Jan 0.5.3 is released to stable version of Jan app: https://github.com/janhq/jan/releases/tag/v0.5.3, feel free to get back to us if any concern

[SimplyProgrammer]

Well, in my case, the problem has wery much persisted also in 0.5.3 :/ I have tried both importing the GPT4ALLs version of the model and also from Jan model browser under name "Llama 3.1 8B !4 Ins" but in both cases it will not start... Its worth mentioning that I have also Llama 3.0 installedm could this be a problem?

[SimplyProgrammer]

I am sorry but it seems like my previous message was not entirely correct. The problem seems to be in the "cortex" API being sneakily uninstalled by the 0.5.3 update. After manually installing it back, it indeed works so my apologies, great job!

[thonore75]

Hi, I have issue too when loading the model "Meta-Llama-3.1-8B-Instruct-128k-Q4_0" In parameters, if I select my both graphic cards, the model is not loading, but in selecting only one, the model is loaded, but still slow (0.76t/s) but the graphic card seems to be used