bug: Nvidia GPU's not being used on windows.

[TashaSkyUp]

Describe the bug I have a 970 and a 3090 installed on my system, but Jan decides to use CPU mode.

Steps to reproduce Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior Jan should use cuda device 0 or allow me to choose which model goes on which device

Screenshots If applicable, add screenshots to help explain your issue.

Environment details

• Operating System: windows 10
• Jan Version: 0.4.3
• Processor: amd 3900x
• RAM: 96gb
• 970 4gb, 3090 24gb

Additional context Add any other context or information that could be helpful in diagnosing the problem.

[MichaelBicker]

I am using a RTX 3060 12gb Variant Nvidia GPU on a second device using Windows 10, I also just get CPU to run automatically. I did not find any documentation or settings to change it. I used the recommended OpenHermes Neural 7B Q4.

So, I also did only run on CPU, GPU / graphic card usage was 0.

Thank you so much for your time and effort on this and helping out : )!

Edit: Forgot to mention, in case it might be important, the maschine was: Ryzen 3500X (yep 3500X, not 3600X), 32gb DDR4 RAM Windows 10 (newest updates)

[DutchyDutch]

Same here!

Windows 11 pro RTX 4080 (16 GB) 32 gigs ram.

CPU is at 83% while the GPU flatlines at 0%

I am sure it's user error but there is no obvious way that I can find to correct this.

Very much appreciate time spend to clear this up!

[bfritscher]

I had to install cuda toolkit and use the experimental build of Jan to make it work. This guide heped https://github.com/janhq/jan/blob/main/USAGE.md

[freelerobot]

Yeah you gotta install Cuda, bc the app doesn't have permission to do this.

Btw, our new nightly build auto-prompts you to do this important step.