bug: Users are forced to use win-cuda-12.0 binary even though the accelerator option is disabled

[hiro-v]

Describe the bug

• Context: https://discord.com/channels/1107178041848909847/1222084065918713909
• User though do not have NVIDIA CUDA (they only have small AMD GPU - Rx580) but they are forced to use win-cuda-12-0 which leads to error.
• He has to rename the win-cuda-12-0 to win-cpu.
• I suspect that there is a problem with the gpu configuration that even he tried the NVIDIA GPU troubleshooting, it's not working.

Steps to reproduce Steps to reproduce the behavior:

1. Go to 'Settings'
2. Click on 'Accelerator' -> It's disabled
3. See error From user When I try to start a model, it tries to use cuda-12 and fails as expected since I don't have NVIDIA:

2024-03-27T03:57:43.994Z [NITRO]::Debug: Spawn nitro at path: C:\Users\me\jan\extensions\@janhq\inference-nitro-extension\dist\bin\win-cuda-12-0\nitro.exe, and args: 1,127.0.0.1,3928
2024-03-27T03:57:44.042Z [NITRO]::Debug: Nitro exited with code: 3221225781
2024-03-27T03:57:44.043Z [NITRO]::Error: child process exited with code 3221225781

Also the UI now sees this too, and it's spitting it out in console: index.html:1 Uncaught (in promise) child process exited with code 3221225781

I renamed win-cpu to win-cuda-12-0 and tried to start the model again - success!

2024-03-27T04:02:16.251Z [NITRO]::CPU informations - 9
2024-03-27T04:02:16.252Z [NITRO]::Debug: Request to kill Nitro
2024-03-27T04:02:16.369Z [NITRO]::Debug: Nitro process is terminated
2024-03-27T04:02:16.936Z [NITRO]::Debug: Spawning Nitro subprocess...
2024-03-27T04:02:16.937Z [NITRO]::Debug: Spawn nitro at path: C:\Users\me\jan\extensions\@janhq\inference-nitro-extension\dist\bin\win-cuda-12-0\nitro.exe, and args: 1,127.0.0.1,3928
...
2024-03-27T04:02:17.333Z [NITRO]::Debug: Nitro is ready
2024-03-27T04:02:17.333Z [NITRO]::Debug: Loading model with params {"ctx_len":4096,"prompt_template":"GPT4 Correct User: {prompt}<|end_of_turn|>GPT4 Correct Assistant:","llama_model_path":"C:\\Users\\me\\jan\\models\\openchat-3.5-7b\\openchat-3.5-0106.Q4_K_M.gguf","user_prompt":"GPT4 Correct User: ","ai_prompt":"<|end_of_turn|>GPT4 Correct Assistant:","cpu_threads":9,"ngl":100}

Expected behavior

• As he does not have eligible accelerators, he should be able to use CPU without the need to tinker around to change the folder name
• The troubleshooting does not work on his side

Screenshots If applicable, add screenshots to help explain your issue.

Environment details

• Operating System: Windows 10
• Jan Version: 0.4.9-344
• Processor: 11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz
• RAM: 68575170560
• Any additional relevant hardware specifics: AMD Rx580

Logs If the cause of the error is not clear, kindly provide your usage logs:

• tail -n 50 ~/jan/logs/app.log if you are using the UI
• tail -n 50 ~/jan/logs/server.log if you are using the local api server Making sure to redact any private information.

Additional context Add any other context or information that could be helpful in diagnosing the problem.

[Andydna2]

can it be related to https://github.com/janhq/jan/issues/2508 ? e.g. failure to load models on Windows 11 Ryzen 7835HS without NVidia ?

[Van-QA]

The issue is reproducible on our side, we will resolve it soon 🙏 Many thanks

[Van-QA]

Tested and the issue was resolved as of Jan v0.4.10-360 ✅ can you help double check the cuda GPU / CPU run mode issue? 🙏