An idea to allow more users to use this locally

[lelapin123]

What about using vicuna is CPU mode and use GPU for what is not Vicuna related ? llama.cpp doesnt use the vram

[Korner83]

Will be super slow, I tested stable diffusion on my 48 thread cpu and it was slower about 100x than my A4000 gpu was. It would take hours at least couple of minutes with a simple prompt. There is a smaller version which can run on 12 GB Vram

[lelapin123]

Alpaca Electron is very decent with 32gb of ram (not vram)

Congrats for your A4000 at $10000/ut

[Korner83]

A4000 is cheap and costs around $900: amazon

I already sold it and bought a 4090 which is 4 times faster.