janhq
Discussion: Validation with nitro on different CPU instruction sets
Problem Link: Check out the issue on GitHub: Issue #2432.
Why This Matters: Jan is designed to work best with newer technology, using something called AVX2 for processing. However, if you're using an older computer, you might not have AVX2, but something like AVX or AVX512 instead. We want to make sure Jan works well for everyone, regardless of their computer's age.
What We've Found: Jan might not work perfectly on every type of computer yet. We're still figuring this out and could use your help to ensure it works for everyone. A new version of Nitro (0.3.15) is out, offering different versions to match your computer's capabilities. Check it out here.
Our Plan:
- March 25 - April 4: We're guiding users through fixing issues by manually updating their Nitro version.
- Future Date: We'll update Jan so it automatically supports all types of computers without needing these manual steps.
How to Manually Fix It:
- Locate Your Jan Folder: You should find it at C:\Users<name>\jan on Windows or ~/jan on Linux.
- Check Your CPU: Look up your CPU (like Intel(R) Core(TM) i7-3770 CPU) online to find its capabilities (e.g., AVX, AVX2, AVX512). Intel's official pages are a good source. 3.Do You Have a Graphics Accelerator? If you have an NVIDIA, AMD, or Intel Arc GPU, it's important to choose the right Nitro version.
- Download the Right Nitro Version: Depending on your system and any accelerators, pick the correct version from the Nitro 0.3.15 release.
- Unzip the Files: Use the guides provided for Windows or Linux to unpack the files.
- Replace the Old Nitro: Move the new Nitro files to the right location in your Jan folder. Make sure it's not nested in another folder.
- Restart Jan: Close and reopen Jan, then check if it's working better.
Let Us Know How It Went: After trying these steps, please share your experience in the GitHub issue comments with details like your Jan version, OS, CPU, and whether you have an accelerator. This helps us make Jan even better for everyone!
I attempted to fix it manually and encountered the same errors. Here are the details:
The folder is located at: ~/jan My CPU is an Intel(R) Core(TM) i7-3740QM (8 cores) @ 3.70 GHz, which supports AVX. My GPU configurations are as follows: GPU 1: AMD Radeon HD 8670A/8670M/8750M / R7 M370 GPU 2: Intel 3rd Gen Core processor Graphics Controller I have 16GB of RAM. I tried two versions: nitro-0.3.15-linux-amd64-avx.tar.gz nitro-0.3.15-linux-amd64-vulkan.tar.gz Both files were unzipped. The folder 'nitro' was moved to ~/jan (this folder did not exist before). Despite these efforts, I encountered the same errors and was unable to start any models Manjaro KDE unstable, Jan as an AppImage the latest
same issure. replaced /home/lost/jan/extensions/@janhq/inference-nitro-extension/dist/bin/linux-cpu/nitro with avx binary from nitro 3.15 and 3.16 with both deb and AppImage all producing the same results
(base) lost@rey:~/jan/extensions/@janhq/inference-nitro-extension/dist/bin/linux-cpu$ jan
Global shortcut registered successfully
[7501:0326/121523.236277:ERROR:viz_main_impl.cc(196)] Exiting GPU process due to errors during initialization
Checking for beta autoupdate feature for deb/rpm distributions
Found package-type: deb
Checking for update
2024-03-26T17:15:23.597Z [SPECS]::Version: 0.4.9-343
2024-03-26T17:15:23.624Z [SPECS]::CPUs: [{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":1200,"times":{"user":105470,"nice":5110,"sys":32610,"idle":6986650,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":2443,"times":{"user":107970,"nice":280,"sys":31320,"idle":6995540,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":2000,"times":{"user":99950,"nice":2460,"sys":29430,"idle":6999910,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":1800,"times":{"user":101820,"nice":110,"sys":29520,"idle":6995090,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":1200,"times":{"user":109860,"nice":330,"sys":41660,"idle":6980290,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":1200,"times":{"user":99970,"nice":10,"sys":29960,"idle":7002330,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":1200,"times":{"user":119660,"nice":320,"sys":33320,"idle":6968220,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":1200,"times":{"user":96890,"nice":3210,"sys":27700,"idle":6996400,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":2100,"times":{"user":103520,"nice":760,"sys":28640,"idle":6999450,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":1200,"times":{"user":89820,"nice":1710,"sys":22180,"idle":7011880,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":2300,"times":{"user":89170,"nice":40,"sys":21080,"idle":7021860,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":2074,"times":{"user":86310,"nice":310,"sys":21000,"idle":7021070,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":1400,"times":{"user":101680,"nice":30,"sys":31620,"idle":7002280,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":3000,"times":{"user":98460,"nice":0,"sys":27920,"idle":7009020,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":1200,"times":{"user":95180,"nice":400,"sys":30780,"idle":7006570,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":3000,"times":{"user":97920,"nice":0,"sys":32180,"idle":7005810,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":1578,"times":{"user":121550,"nice":100,"sys":46320,"idle":6967890,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":1200,"times":{"user":104380,"nice":140,"sys":32800,"idle":6998650,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":2900,"times":{"user":90500,"nice":40,"sys":23390,"idle":7017290,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":2400,"times":{"user":105440,"nice":1600,"sys":25930,"idle":6997410,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":2900,"times":{"user":98110,"nice":190,"sys":24810,"idle":7010280,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":1900,"times":{"user":86110,"nice":60,"sys":20610,"idle":7025530,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":1200,"times":{"user":85820,"nice":640,"sys":18320,"idle":7025990,"irq":0}},{"model":"Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz","speed":1200,"times":{"user":87490,"nice":730,"sys":20860,"idle":7017420,"irq":0}}]
2024-03-26T17:15:23.625Z [SPECS]::Machine: x86_64
2024-03-26T17:15:23.625Z [SPECS]::Endianness: LE
2024-03-26T17:15:23.626Z [SPECS]::Parallelism: 24
2024-03-26T17:15:23.626Z [SPECS]::Free Mem: 63924367360
2024-03-26T17:15:23.627Z [SPECS]::Total Mem: 67471896576
2024-03-26T17:15:23.627Z [SPECS]::OS Version: #111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024
2024-03-26T17:15:23.627Z [SPECS]::OS Platform: linux
2024-03-26T17:15:23.628Z [SPECS]::OS Release: 5.15.0-101-generic
[7581:0326/121524.132949:ERROR:viz_main_impl.cc(196)] Exiting GPU process due to errors during initialization
Update for version 0.4.9-343 is not available (latest version: 0.4.9-343, downgrade is disallowed).
false [ '' ]
Error occurred in handler for 'writeFileSync': TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined
at Object.writeFileSync (node:fs:2314:5)
at /opt/Jan/resources/app.asar/node_modules/@janhq/core/dist/node/index.cjs.js:2147:35 {
code: 'ERR_INVALID_ARG_TYPE'
}
2024-03-26T17:15:55.593Z [NITRO]::CPU informations - 12
2024-03-26T17:15:55.596Z [NITRO]::Debug: Request to kill Nitro
2024-03-26T17:15:55.677Z [NITRO]::Debug: Nitro process is terminated
2024-03-26T17:15:55.678Z [NITRO]::Debug: Spawning Nitro subprocess...
2024-03-26T17:15:55.679Z [NITRO]::Debug: Spawn nitro at path: /home/lost/jan/extensions/@janhq/inference-nitro-extension/dist/bin/linux-cpu/nitro, and args: 1,127.0.0.1,3928
2024-03-26T17:15:55.691Z [NITRO]::Debug: ___ ___ ___
/__/ ___ ___ / /\ / /\
\ \:\ / /\ / /\ / /::\ / /::\
\ \:\ / /:/ / /:/ / /:/\:\ / /:/\:\
_____\__\:\ /__/::\ / /:/ / /:/ \:\ / /:/ \:\
/__/::::::::\ \__\/\:\__ / /::\ /__/:/ /:/___ /__/:/ \__\:\
\ \:\~~\~~\/ \ \:\/\ /__/:/\:\ \ \:\/:::::/ \ \:\ / /:/
\ \:\ ~~~ \__\::/ \__\/ \:\ \ \::/~~~~ \ \:\ /:/
\ \:\ /__/:/ \ \:\ \ \:\ \ \:\/:/
\ \:\ \__\/ \__\/ \ \:\ \ \::/
\__\/
17:15:55.993Z [NITRO]::Debug: Nitro is ready
2024-03-26T17:15:55.994Z [NITRO]::Debug: Loading model with params {"ctx_len":4096,"prompt_template":"<|system|>\n{system_message}<|user|>\n{prompt}<|assistant|>","llama_model_path":"/home/lost/jan/models/tinyllama-1.1b/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf","system_prompt":"<|system|>\n","user_prompt":"<|user|>\n","ai_prompt":"<|assistant|>","cpu_threads":12,"ngl":100}
2024-03-26T17:15:56.767Z [NITRO]::Debug: Nitro exited with code: null
2024-03-26T17:15:58.282Z [NITRO]::Error: Load model failed with error TypeError: fetch failed
2024-03-26T17:15:58.282Z [NITRO]::Error: TypeError: fetch failed
It is not clear to me which version of Nitro works with my Qualcomm Snapdragon Windows devices. These do not support AVX, that is an x86 extension, this is Windows ARM64. Log:
2024-03-27T14:30:23.931Z [APP]::Version: 0.4.9 2024-03-27T14:31:15.623Z [NITRO]::CPU informations - 13 2024-03-27T14:31:15.624Z [NITRO]::Debug: Request to kill Nitro 2024-03-27T14:31:15.656Z [NITRO]::Debug: Nitro process is terminated 2024-03-27T14:31:16.170Z [NITRO]::Debug: Spawning Nitro subprocess... 2024-03-27T14:31:16.172Z [NITRO]::Debug: Spawn nitro at path: C:\Users\ssegan\jan\extensions@janhq\inference-nitro-extension\dist\bin\win-cpu\nitro.exe, and args: 1,127.0.0.1,3928 2024-03-27T14:31:16.278Z [NITRO]::Debug: Nitro exited with code: 3221225501 2024-03-27T14:31:16.279Z [NITRO]::Error: child process exited with code 3221225501
These instructions need a little help. I figured it out. I am using a to Intel 2600K that supports AVX (not avx2) on a Windows 10 system. This nitro fix helped.
To enable or disable AVX instructions you can use this commands in CMD: Enable AVX: bcdedit /set xsavedisable 0 Disable AVX: bcdedit /set xsavedisable 1
- Find this folder C:\Users<USER NAME>\jan\extensions@janhq\inference-nitro-extension\dist\bin\win-cpu. Copy the dlls and exe into another folder just in case.
- Download the correct Nitro build for your system for me its this one nitro-0.3.15-win-amd64-avx.tar.gz from Nitro 0.3.15 release.
- Extract .tar.gz in Windows like this: tar -xvzf C:\Users<user name>\Downloads\nitro-0.3.15-win-amd64-avx.tar.gz -C C:\Users<user name>\jan\extensions@janhq\inference-nitro-extension\dist\bin\win-cpu. OR just download, extract, and manually replace the files in the win-cpu with the new dlls and exe.
- Close and relaunch Jan. You should be able to start your model.
Hi all thank you for trying these out.
Thanks @Snipes01 I will update this to the docs, really appreciate your help Hi @77poker125 Jan does not support Windows with ARM64 at the moment, only x86 one. We will consider this as an option for future plan
Hi @lostsync, I believe your CPU (Intel Xeon) falls into extra options with VNNI - https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#AVX-VNNI,_AVX-IFMA => Will need another option with AVX512_VNNI. I will try to add it as an release.
Hi @Graninius , I believe:
-
GPU 2: Intel 3rd Gen Core processor Graphics Controlleris too small to run LLM, in fact in our test it's worse than normal CPU. -
GPU 1: AMD Radeon HD 8670A/8670M/8750M / R7 M370: These are strange to me, are those integrated card/ previous cards? Could you share more information. Thanks
Hi! I've tried this procedure as suggested.
this is my configuration:
System:
Host: Mint Kernel: 6.5.0-26-generic x86_64 bits: 64
Desktop: Cinnamon 6.0.4 Distro: Linux Mint 21.3 Virginia
Machine:
Type: Desktop Mobo: ASUSTeK model: P8Z68-V PRO v: Rev 1.xx
serial:
this is the result:
15:15:55.646Z [NITRO]::Debug: Loading model with params {"ctx_len":4096,"prompt_template":"<|system|>\n{system_message}<|user|>\n{prompt}<|assistant|>","llama_model_path":"/MY_HOME/jan/models/tinyllama-1.1b/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf","system_prompt":"<|system|>\n","user_prompt":"<|user|>\n","ai_prompt":"<|assistant|>","cpu_threads":4,"ngl":100} 2024-04-04T15:15:55.646Z [NITRO]::Debug: Nitro is ready 2024-04-04T15:15:55.667Z [NITRO]::Debug: Nitro exited with code: null 2024-04-04T15:15:57.174Z [NITRO]::Error: Load model failed with error TypeError: fetch failed 2024-04-04T15:15:57.174Z [NITRO]::Error: TypeError: fetch failed
Hi @Graninius , I believe:
* `GPU 2: Intel 3rd Gen Core processor Graphics Controller` is too small to run LLM, in fact in our test it's worse than normal CPU. * `GPU 1: AMD Radeon HD 8670A/8670M/8750M / R7 M370`: These are strange to me, are those integrated card/ previous cards? Could you share more information. Thanks
GPU2 is an integrated card (intel hd4000) GPU1 is a discrete video card (amd hd8750m, 2gb)
The fix failed for me. Based on my system info I downloaded nitro-0.3.15-linux-amd64-avx.tar.gz from https://github.com/janhq/nitro/releases/tag/v0.3.15 , extracted, and replaced the executable at [myJanDir]/extensions/@janhq/inference-nitro-extension/dist/bin/linux-cpu/nitro.
Jan log confirms this is the binary being used: "2024-04-10T01:11:54.950Z [NITRO]::Debug: Spawn nitro at path: [myJanDir]/extensions/@janhq/inference-nitro-extension/dist/bin/linux-cpu/nitro, and args: 1,127.0.0.1,3928". I checked that permissions on the new and old nitro file matched. Still complains: "2024-04-10T01:11:55.276Z [NITRO]::Debug: Nitro exited with code: null 2024-04-10T01:11:56.787Z [NITRO]::Error: Load model failed with error TypeError: fetch failed"
My system info (emphasis mine):
uname -a ---> Linux 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux
cat /proc/cpuinfo ---> processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 58 model name : Intel(R) Core(TM) i5-3317U CPU @ 1.70GHz stepping : 9 microcode : 0x21 cpu MHz : 800.000 cache size : 3072 KB physical id : 0 siblings : 4 core id : 0 cpu cores : 2 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave [[[[[[---------------->avx<------------------]]]]]] f16c rdrand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts md_clear flush_l1d
nitro-0.3.21-linux-amd64-avx.tar.gz crashes on Intel i5-2500K
traps: DrogonIoLoop[93777] trap invalid opcode ip:6428e8c8d262 sp:743a87bf9910 error:0 in nitro[6428e8a00000+548000]
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 42
model name : Intel(R) Core(TM) i5-2500K CPU @ 3.30GHz
stepping : 7
microcode : 0x2f
cpu MHz : 4429.790
cache size : 6144 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 popcnt tsc_deadline_timer aes xsave avx lahf_lm epb ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid xsaveopt dtherm ida arat pln pts vnmi md_clear flush_l1d
vmx flags : vnmi preemption_timer invvpid ept_x_only flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 6622.26
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
