jan
jan copied to clipboard
janhq
bug: Unable to active any models on Macbook Pro
Hardware:Macbook Pro with 16GB memory
Version:Jan 0.4.6 stable and Jan 0.4.6-271 nightly build
OS:Mac OS Sonoma 14.3.1
Problem:
No model can be activated to work.
It seems like an issue of memory, but I tested TinyChat and Deepseek, problem still.
LOG：
app.log
I installed JAN to a super host with 512GB memory, Ubuntu OS, but the same problem occured. There is still amiss message. App.log here. app.log
Same here. After the latest update, Jan refuses to load any models. Hitting "Start model" just kicks it back to "Inactive". MacBook Pro 16" M1 Pro 16 GB. Worked perfectly fine before the update.
Same here. After the latest update, Jan refuses to load any models. Hitting "Start model" just kicks it back to "Inactive". MacBook Pro 16" M1 Pro 16 GB. Worked perfectly fine before the update.
Please help attach version number you are on, also the app.log. We will investigate further.
@louis-jan sure thing! Jan version 0.4.6. And here's a fresh app.log. Looks like an issue with the llama.cpp version bundled in nitro, or nitro's lookup of the model? It's looking for ...jan/models/mistral-ins-7b-q4/mistral-7b-instruct-v0.2.Q4_K_M.gguf whereas the model locally is named simply mistral-ins-7b-q4 which is also its ID inside model.json (I haven't tampered with any of these files, it's all as-is from when Jan downloaded them).
2024-02-18T23:33:15.351Z [NITRO]::CPU informations - 102024-02-18T23:33:15.353Z [NITRO]::Debug: Request to kill Nitro
2024-02-18T23:33:15.376Z [NITRO]::Debug: Nitro process is terminated
2024-02-18T23:33:15.377Z [NITRO]::Debug: Spawning Nitro subprocess...
2024-02-18T23:33:15.377Z [NITRO]::Debug: Spawn nitro at path: /Users/myuser/jan/extensions/@janhq/inference-nitro-extension/dist/bin/mac-arm64/nitro, and args: 1,127.0.0.1,3928
2024-02-18T23:33:15.687Z [NITRO]::Debug: Nitro is ready
2024-02-18T23:33:15.688Z [NITRO]::Debug: Loading model with params {"ctx_len":4096,"prompt_template":"[INST] {prompt} [/INST]","llama_model_path":"/Users/myuser/jan/models/mistral-ins-7b-q4/mistral-7b-instruct-v0.2.Q4_K_M.gguf","user_prompt":"[INST] ","ai_prompt":" [/INST]","cpu_threads":10,"ngl":100}
2024-02-18T23:33:15.699Z [NITRO]::Debug: ## MY EDIT: VERY LONG STRING FULL OF [94m: [93m: etc etc etc… ##
20240218 23:33:15.403396 UTC 2110942 INFO Nitro version: - main.cc:50
20240218 23:33:15.403542 UTC 2110942 INFO Server started, listening at: 127.0.0.1:3928 - main.cc:54
20240218 23:33:15.403543 UTC 2110942 INFO Please load your model - main.cc:55
20240218 23:33:15.403544 UTC 2110942 INFO Number of thread is:10 - main.cc:62
{"timestamp":1708299195,"level":"INFO","function":"loadModelImpl","line":561,"message":"system info","n_threads":10,"total_threads":10,"system_info":"AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | "}
2024-02-18T23:33:15.700Z [NITRO]::Error: llama_model_load: error loading model: failed to open /Users/myuser/jan/models/mistral-ins-7b-q4/mistral-7b-instruct-v0.2.Q4_K_M.gguf: No such file or directory
2024-02-18T23:33:15.700Z [NITRO]::Error: llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model '/Users/myuser/jan/models/mistral-ins-7b-q4/mistral-7b-instruct-v0.2.Q4_K_M.gguf'
2024-02-18T23:33:15.701Z [NITRO]::Debug: {"timestamp":1708299195,"level":"ERROR","function":"load_model","line":560,"message":"unable to load model","model":"/Users/myuser/jan/models/mistral-ins-7b-q4/mistral-7b-instruct-v0.2.Q4_K_M.gguf"}
20240218 23:33:15.700512 UTC 2110961 ERROR Error loading the model - llamaCPP.cc:565
2024-02-18T23:33:15.708Z [NITRO]::Debug: Load model success with response {}
2024-02-18T23:33:15.711Z [NITRO]::Debug: Validate model state with response 200
2024-02-18T23:33:15.713Z [NITRO]::Debug: Validate model state failed with response "OK"
2024-02-18T23:33:15.714Z [NITRO]::Error: Validate model status failed
Heads up: deleting the model through Jan and redownloading it again seem to have fixed the issue on my end. I can stop/start the model without any issues now! 👍 Prob paths got nixed in the update (?).
Heads up: deleting the model through Jan and redownloading it again seem to have fixed the issue on my end. I can stop/start the model without any issues now! 👍 Prob paths got nixed in the update (?).
Thank you for the great find. Please try again with the latest nightly build if possible. The paths fix has not been released yet (0.4.7).
Heads up: deleting the model through Jan and redownloading it again seem to have fixed the issue on my end. I can stop/start the model without any issues now! 👍 Prob paths got nixed in the update (?).
@t7aliang please help try factory reset (app) on the latest nightly build.
Heads up: deleting the model through Jan and redownloading it again seem to have fixed the issue on my end. I can stop/start the model without any issues now! 👍 Prob paths got nixed in the update (?).
@t7aliang please help try factory reset (app) on the latest nightly build.
Updated to the latest nightly build version, factory reseted, models downloaded again, PROBLEM STILL. Here is the log. Maybe I should .... here is China mainland.
No matter whether I use a MacBook Pro, Ubuntu Linux, or Windows 11, I can't activate any models. So, is the problem with my network environment? However, isn't JAN a localized AI terminal? Are there any special network connections required during the program's initialization or the model's activation process?
I switched to my mobile data network and downloaded a 600Mb model, which I finally managed to activate successfully. However, even on the mobile network, I cannot activate and start other models that have already been downloaded; only the models downloaded via my mobile network can be successfully activated. Oddly enough, the model that I was able to activate works fine under any network condition or even without any network connection. It's strange that the network used to download the model determines whether it can be activated or not. Additionally, I tried downloading a Gemma-2b model from HuggingFace and placed it in the corresponding directory within the models directory. It appears in JAN as having been added, but I still can't activate and use it.
So, how can this issue be resolved? — The network used to download the model determines whether it can be activated.
@t7aliang, please try our latest nightly build with Gemma support.
I was able to download and activate Gemma 7B normally in the latest nightly build version (v0.4.7-304), and this time I could also activate the Gemma 2B model previously downloaded from Hugging Face, but Mistral and OpenChat3.5 still cannot be activated.
@t7aliang, please try our latest nightly build with Gemma support.
I was able to download and activate Gemma 7B normally in the latest nightly build version (v0.4.7-304), and this time I could also activate the Gemma 2B model previously downloaded from Hugging Face, but Mistral and OpenChat3.5 still cannot be activated.
Great to hear that you can run Gemma models! I believe there might be a RAM-related issue. Could you please reduce the context length to 1024 or 2048? You can do this in the right panel, under Thread Settings > Engine Parameters > Context Length.
@t7aliang, please try our latest nightly build with Gemma support.
I was able to download and activate Gemma 7B normally in the latest nightly build version (v0.4.7-304), and this time I could also activate the Gemma 2B model previously downloaded from Hugging Face, but Mistral and OpenChat3.5 still cannot be activated.
Great to hear that you can run Gemma models! I believe there might be a RAM-related issue. Could you please reduce the context length to 1024 or 2048? You can do this in the right panel, under Thread Settings > Engine Parameters > Context Length.
I have set the context length to 2048, using the Gemma 7B model, and simply tested its code generation capabilities in both Chinese and English. It generated many characters that are incomprehensible. For example, like this: | __.--' _)|. And like this: # Paste the extracted data into a new document pyautogui.paste() A I E T R C H M P L Z V B X N Y K.
In my case, disabling GPU Acceleration in Advance Settings helped. Prior to that, I couldn't activate ANY model. Hardware: ASUS laptop with NVIDIA GeForce GTX 1660Ti and 40GB memory Version:Jan 0.4.9 OS: Windows 10 22H2 (194045.4170)
hi @WainLai, can you share with us:
- Jan app version
- App.log if the nitro is returning 3221225781 error code, can you try this https://jan.ai/troubleshooting/undefined-issue/ and see if you can start nitro manually? After that, if you are having problem starting the Nitro, please try this: https://jan.ai/troubleshooting/#troubleshooting-nvidia-gpu can you try installing all these drivers from here?
Yes, it was 3221225781. I installed a new driver and CUDA, and everything is hunky dory. THANKS!