"Illegal Instruction" error when converting 7B model to ggml FP16 format (Raspberry Pi 4, 8GB, Raspberry Pi OS, 64-bit)

[lesp]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [ /] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• [ /] I carefully followed the README.md.
• [ /] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• [ /] I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

I expected the command to convert the 7B model to ggml FP16 format

Current Behavior

Illegal instruction error

les@raspberrypi:~/llama.cpp $ python3 convert-pth-to-ggml.py models/7B/ 1
Illegal instruction

Environment and Context

Raspberry Pi 4 8GB Latest Raspberry Pi OS 64-bit, fully updated

• Physical (or virtual) hardware you are using, e.g. for Linux: $lscpu Architecture: aarch64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 Vendor ID: ARM Model: 3 Model name: Cortex-A72 Stepping: r0p3 CPU max MHz: 1800.0000 CPU min MHz: 600.0000 BogoMIPS: 108.00 L1d cache: 128 KiB L1i cache: 192 KiB L2 cache: 1 MiB Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec store bypass: Vulnerable Vulnerability Spectre v1: Mitigation; __user pointer sanitization Vulnerability Spectre v2: Vulnerable Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Flags: fp asimd evtstrm crc32 cpuid

• Operating System, e.g. for Linux:

$ uname -a Linux raspberrypi 5.15.84-v8+ #1613 SMP PREEMPT Thu Jan 5 12:03:08 GMT 2023 aarch64 GNU/Linux

• SDK version, e.g. for Linux:

$ python3 --version
Python 3.9.2
$ make --version
GNU Make 4.3
Built for aarch64-unknown-linux-gnu

$ g++ --version

g++ (Debian 10.2.1-6) 10.2.1 20210110

Failure Information (for bugs)

Please help provide information about the failure if this is a bug. If it is not a bug, please remove the rest of this template.

Steps to Reproduce

Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.

1. Cloned the repo.
2. Changed directory to llama.cpp.
3. Used make.
4. Downloaded the 7B model and copied it to /models/7B.
5. Installed the Python modules torch numpy sentencepiece.
6. Used convert-pth-to-ggml.py models/7B/ 1 to convert the model to ggml FP16 format.
7. Received the illegal instruction error after a few seconds.

[mqy]

Should due to lack of memory. convert-pth-to-ggml.py loads the entire 13GB models/7B/consolidated.00.pth as pytorch model.

Another similar issue https://github.com/ggerganov/llama.cpp/issues/402

[lesp]

Confirmed that it was a RAM issue. I managed to complete the project on my AMD Ryzen 5600X with 32GB RAM. Copied the files from /models over to the Pi and all worked well. Thanks for the advice.