bug: Model cannot start on Mac M3

[hiro-v]

Describe the bug

• Model cannot start on Mac M3 llama3-8b-instruct

Steps to reproduce Steps to reproduce the behavior:

1. Download model
2. Click Use
3. Chat
4. It shows model cannot load
5. Check the log and see the error

ERROR Could not load engine: Could not load library "./engines/cortex.llamacpp/libengine.dylib"

Expected behavior

• Model starts and can chat normally

Screenshots If applicable, add screenshots to help explain your issue.

Environment details

• Operating System: MacOS 14.4.1
• Jan Version: Jan v0.4.12-435
• Processor: Apple M3
• RAM: 32GB RAM
• Any additional relevant hardware specifics: [e.g., Graphics card, SSD/HDD]

Logs

2024-05-17T03:26:09.120Z [SPECS]::Version: 0.4.12-4352024-05-17T03:26:09.121Z [SPECS]::CPUs: [{"model":"Apple M3 Pro","speed":24,"times":{"user":188596440,"nice":0,"sys":125261940,"idle":889957730,"irq":0}},{"model":"Apple M3 Pro","speed":24,"times":{"user":159653960,"nice":0,"sys":100688510,"idle":949465300,"irq":0}},{"model":"Apple M3 Pro","speed":24,"times":{"user":118884900,"nice":0,"sys":68591080,"idle":1030493130,"irq":0}},{"model":"Apple M3 Pro","speed":24,"times":{"user":82693540,"nice":0,"sys":43202680,"idle":1098255630,"irq":0}},{"model":"Apple M3 Pro","speed":24,"times":{"user":62060280,"nice":0,"sys":29563860,"idle":1136126430,"irq":0}},{"model":"Apple M3 Pro","speed":24,"times":{"user":49663330,"nice":0,"sys":22410700,"idle":1157780260,"irq":0}},{"model":"Apple M3 Pro","speed":24,"times":{"user":41871120,"nice":0,"sys":10771500,"idle":1180387850,"irq":0}},{"model":"Apple M3 Pro","speed":24,"times":{"user":40673060,"nice":0,"sys":10549550,"idle":1182069780,"irq":0}},{"model":"Apple M3 Pro","speed":24,"times":{"user":40293510,"nice":0,"sys":10452250,"idle":1182733880,"irq":0}},{"model":"Apple M3 Pro","speed":24,"times":{"user":40033180,"nice":0,"sys":10435860,"idle":1183089040,"irq":0}},{"model":"Apple M3 Pro","speed":24,"times":{"user":39877320,"nice":0,"sys":10460290,"idle":1183293120,"irq":0}}]
2024-05-17T03:26:09.121Z [SPECS]::Endianness: LE
2024-05-17T03:26:09.121Z [SPECS]::Free Mem: 1661222912
2024-05-17T03:26:09.121Z [SPECS]::Total Mem: 38654705664
2024-05-17T03:26:09.121Z [SPECS]::OS Version: Darwin Kernel Version 23.4.0: Fri Mar 15 00:12:25 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6030
2024-05-17T03:26:09.121Z [SPECS]::OS Platform: darwin
2024-05-17T03:26:09.121Z [SPECS]::Machine: arm64
2024-05-17T03:26:09.121Z [SPECS]::Parallelism: 11
2024-05-17T03:26:09.121Z [CORTEX]::CPU information - 11
2024-05-17T03:26:09.121Z [CORTEX]::Debug: Request to kill cortex
2024-05-17T03:26:09.121Z [SPECS]::OS Release: 23.4.0
2024-05-17T03:26:09.138Z [CORTEX]::Debug: cortex process is terminated
2024-05-17T03:26:09.138Z [CORTEX]::Debug: Spawning cortex subprocess...
2024-05-17T03:26:09.139Z [APP]::/Users/hiro/jan/extensions/@janhq/inference-cortex-extension/dist/bin/mac-arm64
2024-05-17T03:26:09.139Z [CORTEX]::Debug: Spawn cortex at path: /Users/hiro/jan/extensions/@janhq/inference-cortex-extension/dist/bin/mac-arm64/cortex-cpp, and args: 1,127.0.0.1,3928
2024-05-17T03:26:09.559Z [CORTEX]::Debug: cortex is ready
2024-05-17T03:26:09.559Z [CORTEX]::Debug: Loading model with params {"cpu_threads":11,"ctx_len":2048,"prompt_template":"<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{system_message}<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n","llama_model_path":"/Users/hiro/jan/models/llama3-8b-instruct/Meta-Llama-3-8B-Instruct-Q4_K_M.gguf","ngl":32,"system_prompt":"<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n","user_prompt":"<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n","ai_prompt":"<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n","model":"llama3-8b-instruct"}
2024-05-17T03:26:09.563Z [CORTEX]::Debug: 20240517 03:26:09.512154 UTC 18118032 INFO  cortex-cpp version: default_version - main.cc:50
20240517 03:26:09.513108 UTC 18118032 INFO  cortex.llamacpp version: 0.1.2 - main.cc:55
20240517 03:26:09.513110 UTC 18118032 INFO  Server started, listening at: 127.0.0.1:3928 - main.cc:58
20240517 03:26:09.513112 UTC 18118032 INFO  Please load your model - main.cc:59
20240517 03:26:09.513120 UTC 18118032 INFO  Number of thread is:11 - main.cc:66
20240517 03:26:09.563100 UTC 18118054 ERROR Could not load engine: Could not load library "./engines/cortex.llamacpp/libengine.dylib"
dlopen(./engines/cortex.llamacpp/libengine.dylib, 0x0006): tried: './engines/cortex.llamacpp/libengine.dylib' (relative path not allowed in hardened program), '/System/Volumes/Preboot/Cryptexes/OS./engines/cortex.llamacpp/libengine.dylib' (no such file), '/usr/lib/./engines/cortex.llamacpp/libengine.dylib' (no such file, not in dyld cache), './engines/cortex.llamacpp/libengine.dylib' (relative path not allowed in hardened program) - server.cc:152

2024-05-17T03:26:09.568Z [CORTEX]::Debug: Load model success with response {}
2024-05-17T03:26:09.570Z [CORTEX]::Debug: Validate model state failed with response "Conflict"
2024-05-17T03:26:09.570Z [CORTEX]::Debug: Validate model state with response 409
2024-05-17T03:26:09.570Z [CORTEX]::Error: Validate model status failed

Additional context Add any other context or information that could be helpful in diagnosing the problem.