GPU vs CPU performance?

[kasfictionlive]

Does anyone have any benchmarks or rough numbers for how much faster the GPU version is? ie is it 2x 4x or what?

[TripleKiller666]

Using this Repo on a Ryzen 3700X, and it generates Text about 5 Seconds after typing in Text. I´m using the Standart Quantized Model, and Unfiltered Model, and both are about the same speed.

[garanews]

will be happy to do some test but currently I am stuck with:

----> 2 from nomic.gpt4all import GPT4AllGPU

ImportError: cannot import name 'GPT4AllGPU' from 'nomic.gpt4all' (/usr/local/lib/python3.9/dist-packages/nomic/gpt4all/init.py)

[spacemining]

from nomic.gpt4all import GPT4AllGPU

do this: from nomic.gpt4all.gpt4all import GPT4AllGPU

[cocobeach]

will be happy to do some test but currently I am stuck with:

----> 2 from nomic.gpt4all import GPT4AllGPU

ImportError: cannot import name 'GPT4AllGPU' from 'nomic.gpt4all' (/usr/local/lib/python3.9/dist-packages/nomic/gpt4all/init.py)

Just do 'pip install nomic' and go CPU, put the bin file in the chat folder and run the command in the terminal, it's more than enough, unless you start talking about something woke and contradict it, then it will eventually crash and quit, so you just restart it. In my view the point of this AI is not so much to chat with it, as it is it is inferior to gpt3.5, but with some clever finetuning it could become a very good specialised tool, probably maybe. At that point you might need the GPU to make most of it.

[garanews]

thanks @spacemining , in that way it works. So @kasfictionlive and @TripleKiller666 if you can write down here the steps to do a benchmark, I will be happy to do it!

[s5s00000]

Hi I have a Windows 10 PC with 8GB RAM and i5-5200 CPU and a NVIDIA GForce 840M. How to let the AI run with the GPU instead of CPU? It's awfully slow. And how do I store my conversations?

[niansa]

Stale, please open a new issue if this is still relevant.