llama
llama copied to clipboard
meta-llama
This is how to run it on Shadow PC 😎
Hello, I got the 7B model to work on a Shadow PC with just 12GB RAM and 16GB P5000 GPU 😲. (This is equivalent to about a Nvidia 1080)
If anyone wants a referral code I think you get money off your first month you can use this one: D4D1803
It took precisely 2 minutes to load the model. Then it took 19 seconds for each subsequent 256 tokens.
You can use my updated example.py file here
I modified it so you can type in new prompts without having to reload the model.
I am going to be researching ways to make it use even less RAM so it will load the model faster. Such as this method.
Here is a screenshot:
TIP: Close as many other programs as you can to free up RAM. Especially things like browsers and even drobox. The more RAM you free the faster the model will load. After the model is loaded the RAM is freed again so this won't affect generation times.
It's kind of neat to be able to run your own little "brain". 😁
I got this error log:
prompt = str(input())
OSError: [Errno 9] Bad file descriptor
I got this error log:
prompt = str(input()) OSError: [Errno 9] Bad file descriptor
That's strange. Well, I can tell you I ran it through the command prompt on Windows 10 if that's any help. That line in question just gets the new prompt from the user so I don't know why that would cause an error. 🤔 Feel free to modify the python code for your own purposes.
Hi @pauldog
Hi, I tried with your script but still got torch.cuda.OutOfMemoryError. My PC had 16GB RAM and 12GBVRAM GPU (rtx 3060) Anyway, My PC had 2 rtx 3060, not sure if I can run 7B model using both GPUs.
Thanks
Hi @pauldog
Hi, I tried with your script but still got torch.cuda.OutOfMemoryError. My PC had 16GB RAM and 12GBVRAM GPU (rtx 3060) Anyway, My PC had 2 rtx 3060, not sure if I can run 7B model using both GPUs.
Thanks
Hi, yes 12GB VRAM is not enough to run it as it is. Have a look at #79 for some forks where people have for it to run on smaller GPUs.
You are lucky, only a few of shadow pcs have got the p5000. Most of them have a 8gb 1080 (inlcuded mine 😢)
You are lucky, only a few of shadow pcs have got the p5000. Most of them have a 8gb 1080 (inlcuded mine 😢)
Maybe it's a UK thing.