llama.cpp icon indicating copy to clipboard operation
llama.cpp copied to clipboard

convert the 7B model to ggml FP16 format fails on RPi 4B

Open davidrutland opened this issue 1 year ago • 8 comments

Everything's OK until this step

python3 convert-pth-to-ggml.py models/7B/ 1 {'dim': 4096, 'multiple_of': 256, 'n_heads': 32, 'n_layers': 32, 'norm_eps': 1e-06, 'vocab_size': 32000} n_parts = 1 Processing part 0 Killed

models/7B/ggml-model-f16.bin isn't created

davidrutland avatar Mar 14 '23 17:03 davidrutland

Ah. Got it. I think I need a bigger SD card.

davidrutland avatar Mar 14 '23 17:03 davidrutland

I have the same problem, the ram increases until it is completely filled and that's as far as it goes

JavSensei avatar Mar 14 '23 17:03 JavSensei

I'm pretty sure you need memory for both the original fp16 model + the converted model. A RPi 4 (even with 8 GB of RAM) isn't going to have enough.

Ronsor avatar Mar 14 '23 21:03 Ronsor

Would it be okay to share a torrent of converted models? Would be a smaller download

BrianSemiglia avatar Mar 14 '23 21:03 BrianSemiglia

There are indeed torrents in places like 4chan. I personally don't think model weights can even be copyrighted, but the repo maintainer probably doesn't want to risk it, otherwise there would likely already be a torrent link provided.

Ronsor avatar Mar 14 '23 21:03 Ronsor

convert the 7B model need 16G RAM.

xiliuya avatar Mar 14 '23 23:03 xiliuya

If you're having memory trouble you should make a swapfile. I only have 8GB of memory, and I'm going back and forth between tasks, so I just wrote a script that makes it easy to create and destroy swapfiles.

https://github.com/apaz-cli/Scripts/blob/master/swapcreate https://github.com/apaz-cli/Scripts/blob/master/swapdestroy

Or you can find instructions online.

apaz-cli avatar Mar 15 '23 00:03 apaz-cli

Ah. Got it. I think I need a bigger SD card.

you can convert it on any computer then just copy it over.

G2G2G2G avatar Mar 15 '23 08:03 G2G2G2G

This looks to be a RAM issue. Closing.

gjmulder avatar Mar 15 '23 21:03 gjmulder