llama.cpp
llama.cpp copied to clipboard
Error while converting to ggml.py format
After running the command: "python3 convert-pth-to-ggml.py /Users/tanish.shah/llama.cpp/models/7B/ 1" Error with sentencepiece:
Traceback (most recent call last):
File "/Users/tanish.shah/llama.cpp/convert-pth-to-ggml.py", line 75, in <module>
tokenizer = sentencepiece.SentencePieceProcessor(fname_tokenizer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tanish.shah/llama.cpp/env/lib/python3.11/site-packages/sentencepiece/__init__.py", line 447, in Init
self.Load(model_file=model_file, model_proto=model_proto)
File "/Users/tanish.shah/llama.cpp/env/lib/python3.11/site-packages/sentencepiece/__init__.py", line 905, in Load
return self.LoadFromFile(model_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tanish.shah/llama.cpp/env/lib/python3.11/site-packages/sentencepiece/__init__.py", line 310, in LoadFromFile
return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Internal: /private/var/folders/rz/kp316btj1lgdlrcgdq7spdg80000gn/T/pip-install-rga_wjtr/sentencepiece_ef6dee7cfe954a50b06d772071b44d95/sentencepiece/src/sentencepiece_processor.cc(1102) [model_proto->ParseFromArray(serialized.data(), serialized.size())]
Please check whether you downloaded the model weights correctly: https://github.com/ggerganov/llama.cpp/issues/238