dalai memory requirements

maybe I'm an idiot but I have to ask is below memory requirements for cpu or gpu ram?

Runs on most modern computers. Unless your computer is very very old, it should work.

According to https://github.com/ggerganov/llama.cpp/issues/13, here are the memory requirements:

7B => ~4 GB 13B => ~8 GB 30B => ~16 GB 65B => ~32 GB

Mar 22 '23 20:03 mishav78

I'm running the 30B alpaca and my memory usage is roughly 78% of my 32GB RAM while in use.

Mar 22 '23 20:03 trevtravtrev

cpu or gpu ram?

Mar 22 '23 20:03 mishav78

My PC memory/RAM. It also uses your CPU. As far as I know there are no current configurable settings to use GPU.

Mar 22 '23 20:03 trevtravtrev

will it work faster with a gpu?

Mar 22 '23 20:03 mishav78

Please see my above edited comment.

Mar 22 '23 20:03 trevtravtrev

very strange. Don't these models usually use gpus?

Mar 22 '23 20:03 mishav78

x

Mar 22 '23 21:03 mishav78

very strange. Don't these models usually use gpus?

This project is using llama.cpp/alpaca.cpp which "Runs on the CPU"

https://github.com/antimatter15/alpaca.cpp#getting-started-30b

Mar 23 '23 04:03 DustinBrett

very strange. Don't these models usually use gpus?

To train them not to run them.

Mar 23 '23 04:03 trevtravtrev

does it work as good as chatgpt? Or close?

Mar 23 '23 04:03 mishav78

I'd say 30B is closing in at about 80% of chat gpt 3.5. 7B/13B maybe 60%+.

Mar 23 '23 05:03 trevtravtrev

I'd say 30B is closing in at about 80% of chat gpt 3.5. 7B/13B maybe 60%+.

I'd be interested to know what prompts you've tried and what parameter (temperature, etc) values you have.

For me even 30B feels like 10% of what I see with ChaGPT 3.5.

Mar 23 '23 06:03 DustinBrett