epic: Jan supports multiple CPU/GPU Hardware optimizations

[tikikun]

Problem

Windows users encounter elevated CPU usage (ex: #887), prompting the need for hardware optimizations in Nitro/Jan across various Windows architectures (CPU, GPU).

Success Criteria

• Achieve compatibility for Jan on Windows with 16GB of RAM, accommodating systems lacking a GPU.
• Enable Jan to operate seamlessly on Windows systems featuring Nvidia GPUs.

Out-of-scope

Not support Intel, AMD GPU for now, although we still detect these device.

Tasklist

Design & FE

• [x] #1646

Implementation

• [x] #286
• [x] #912
• [x] #913
• [ ] #915
• [x] #911
• [x] #914
• [x] #1730

Additional context

• Edge inferencing on windows is very important, Intel and Nvidia hardware having multiple optimizations that can be tuned to make LLM inference possible on windows.
• https://www.intel.com/content/www/us/en/developer/tools/oneapi/overview.html

[tikikun]

https://www.npmjs.com/package/systeminformation

[tikikun]

Intel optimization: Instructions set: AVX2 - Consumer grade AVX512 - Before gen 12 or Consumer grade AVX_VNNI - Consumer grade with OneAPI AVX512_VNNI - Server grade

[tikikun]

Windows WSL https://github.com/janhq/jan/issues/912

oneAPI for Windows https://github.com/janhq/jan/issues/911

AMD GPU for Windows https://github.com/janhq/jan/issues/913

AMD CPU for Windows https://github.com/janhq/jan/issues/914

Intel GPU (maybe crossing with oneAPI) https://github.com/janhq/jan/issues/915

[dan-menlo]

@tikikun I've renamed this Epic and added more detail to the "Top" post, please take a look.

I will be creating a separate "Jan should work on Windows" epic, that will be more Product-focused (there will be some overlap)

[hantran-co]

Archive the original comment of Alan:

---

Problem

• Windows users experience 100% CPU usage (#887)
• Nitro/Jan needs to support hardware optimizations for different Windows architectures (CPU, GPU)

Success Criteria

• Jan can run on Windows with 16gb of RAM, no GPU (both Intel and AMD processors)
• Jan can run on Windows with Intel, AMD, Nvidia GPUs

Tasks

• [x] #912
• [x] #911
• [x] #913
• [x] #914
• [x] #915

Additional context

• Edge inferencing on windows is very important, Intel and Nvidia hardware having multiple optimizations that can be tuned to make LLM inference possible on windows.
• https://www.intel.com/content/www/us/en/developer/tools/oneapi/overview.html

[dan-menlo]

@Van-QA to decompose this into the engineering: Jan Inference Layer roadmap

[Van-QA]

Only 1 feat and 1 epic left, I will close this epic once the remaining ticket #915 completed. On the other hand, I will follow up with the epic #913 in the engineering: Jan Inference Layer roadmap

[Van-QA]

as discussed with @hiro-v, the remaining feat #915 is nice to have, and will handled in the follow up epic #2092