Error when converting safe tensors to gguf

[ieatbeansbruh]

M3 pro MacBook Pro

error when converting safe tensors to gguf

(base) colin@colins-mbp llama.cpp % python convert.py --outtype f16 --ctx 32768 models/Smaug Traceback (most recent call last): File "/Users/colin/llama.cpp/convert.py", line 1483, in main() File "/Users/colin/llama.cpp/convert.py", line 1419, in main model_plus = load_some_model(args.model) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/colin/llama.cpp/convert.py", line 1269, in load_some_model raise Exception(f"Can't find model in directory {path}") Exception: Can't find model in directory models/Smaug

[phymbert]

@ieatbeansbruh Hi, be sure you have downloaded all model files in folder models/Smaug. Especially it looks there is no model-00001-of-*.safetensors. Please confirm with ls -al models/Smaug.

[dspasyuk]

@ieatbeansbruh were you able to convert Smaug model to gguf? I am getting this error: python3 convert.py --outtype f16 --ctx 32768 ../../models/models/abacusai_Smaug-72B-v0.1/ Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00001-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00001-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00002-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00003-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00004-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00005-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00006-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00007-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00008-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00009-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00010-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00011-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00012-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00013-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00014-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00015-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00016-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00017-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00018-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00019-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00020-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00021-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00022-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00023-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00024-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00025-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00026-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00027-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00028-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00029-of-00030.safetensors Loading model file ../../models/models/abacusai_Smaug-72B-v0.1/model-00030-of-00030.safetensors params = Params(n_vocab=152064, n_embd=8192, n_layer=80, n_ctx=32768, n_ff=24576, n_head=64, n_head_kv=64, n_experts=None, n_experts_used=None, f_norm_eps=1e-06, rope_scaling_type=None, f_rope_freq_base=1000000, f_rope_scale=None, n_orig_ctx=None, rope_finetuned=None, ftype=<GGMLFileType.MostlyF16: 1>, path_model=PosixPath('../../models/models/abacusai_Smaug-72B-v0.1')) Found vocab files: {'tokenizer.model': None, 'vocab.json': PosixPath('../../models/models/abacusai_Smaug-72B-v0.1/vocab.json'), 'tokenizer.json': PosixPath('../../models/models/abacusai_Smaug-72B-v0.1/tokenizer.json')} Loading vocab file '../../models/models/abacusai_Smaug-72B-v0.1/vocab.json', type 'spm' Traceback (most recent call last): File "/home/denis/CODE/lama/llama.cpp/convert.py", line 1483, in main() File "/home/denis/CODE/lama/llama.cpp/convert.py", line 1451, in main vocab, special_vocab = vocab_factory.load_vocab(args.vocab_type, model_parent_path) File "/home/denis/CODE/lama/llama.cpp/convert.py", line 1336, in load_vocab vocab = SentencePieceVocab( File "/home/denis/CODE/lama/llama.cpp/convert.py", line 394, in init self.sentencepiece_tokenizer = SentencePieceProcessor(str(fname_tokenizer)) File "/home/denis/.local/lib/python3.10/site-packages/sentencepiece/init.py", line 447, in Init self.Load(model_file=model_file, model_proto=model_proto) File "/home/denis/.local/lib/python3.10/site-packages/sentencepiece/init.py", line 905, in Load return self.LoadFromFile(model_file) File "/home/denis/.local/lib/python3.10/site-packages/sentencepiece/init.py", line 310, in LoadFromFile return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg) RuntimeError: Internal: src/sentencepiece_processor.cc(1101) [model_proto->ParseFromArray(serialized.data(), serialized.size())]

[hackdefendr]

Confirmed. I am seeing this same behavior with the latest Starcoder2 model.

~/devel/llama.cpp$ python3 convert.py --pad-vocab --skip-unknown --outtype f16 /home/jpop/devel/models/bigcode_starcoder2-15b
Loading model file /home/jpop/devel/models/bigcode_starcoder2-15b/model-00001-of-00014.safetensors
Loading model file /home/jpop/devel/models/bigcode_starcoder2-15b/model-00001-of-00014.safetensors
Loading model file /home/jpop/devel/models/bigcode_starcoder2-15b/model-00002-of-00014.safetensors
Loading model file /home/jpop/devel/models/bigcode_starcoder2-15b/model-00003-of-00014.safetensors
Loading model file /home/jpop/devel/models/bigcode_starcoder2-15b/model-00004-of-00014.safetensors
Loading model file /home/jpop/devel/models/bigcode_starcoder2-15b/model-00005-of-00014.safetensors
Loading model file /home/jpop/devel/models/bigcode_starcoder2-15b/model-00006-of-00014.safetensors
Loading model file /home/jpop/devel/models/bigcode_starcoder2-15b/model-00007-of-00014.safetensors
Loading model file /home/jpop/devel/models/bigcode_starcoder2-15b/model-00008-of-00014.safetensors
Loading model file /home/jpop/devel/models/bigcode_starcoder2-15b/model-00009-of-00014.safetensors
Loading model file /home/jpop/devel/models/bigcode_starcoder2-15b/model-00010-of-00014.safetensors
Loading model file /home/jpop/devel/models/bigcode_starcoder2-15b/model-00011-of-00014.safetensors
Loading model file /home/jpop/devel/models/bigcode_starcoder2-15b/model-00012-of-00014.safetensors
Loading model file /home/jpop/devel/models/bigcode_starcoder2-15b/model-00013-of-00014.safetensors
Loading model file /home/jpop/devel/models/bigcode_starcoder2-15b/model-00014-of-00014.safetensors
params = Params(n_vocab=49152, n_embd=6144, n_layer=40, n_ctx=16384, n_ff=24576, n_head=48, n_head_kv=4, n_experts=None, n_experts_used=None, f_norm_eps=1e-05, rope_scaling_type=None, f_rope_freq_base=100000, f_rope_scale=None, n_orig_ctx=None, rope_finetuned=None, ftype=<GGMLFileType.MostlyF16: 1>, path_model=PosixPath('/home/jpop/devel/models/bigcode_starcoder2-15b'))
Found vocab files: {'tokenizer.model': None, 'vocab.json': PosixPath('/home/jpop/devel/models/bigcode_starcoder2-15b/vocab.json'), 'tokenizer.json': PosixPath('/home/jpop/devel/models/bigcode_starcoder2-15b/tokenizer.json')}
Loading vocab file '/home/jpop/devel/models/bigcode_starcoder2-15b/vocab.json', type 'spm'
Traceback (most recent call last):
  File "/home/jpop/devel/llama.cpp/convert.py", line 1483, in <module>
    main()
  File "/home/jpop/devel/llama.cpp/convert.py", line 1451, in main
    vocab, special_vocab = vocab_factory.load_vocab(args.vocab_type, model_parent_path)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jpop/devel/llama.cpp/convert.py", line 1336, in load_vocab
    vocab = SentencePieceVocab(
            ^^^^^^^^^^^^^^^^^^^
  File "/home/jpop/devel/llama.cpp/convert.py", line 394, in __init__
    self.sentencepiece_tokenizer = SentencePieceProcessor(str(fname_tokenizer))
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jpop/.local/lib/python3.11/site-packages/sentencepiece/__init__.py", line 447, in Init
    self.Load(model_file=model_file, model_proto=model_proto)
  File "/home/jpop/.local/lib/python3.11/site-packages/sentencepiece/__init__.py", line 905, in Load
    return self.LoadFromFile(model_file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jpop/.local/lib/python3.11/site-packages/sentencepiece/__init__.py", line 310, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Internal: src/sentencepiece_processor.cc(1101) [model_proto->ParseFromArray(serialized.data(), serialized.size())]

UPDATE (Mar 2nd 2024 8:11PM)

The default vocab type is smp, for my issue at least, you have to force set the vocab-type to hfft.

Like so:

python3 convert.py --outtype f16 --vocab-type hfft models/bigcode_starcoder2-15b

[countzero]

@ieatbeansbruh, @phymbert, @dspasyuk & @hackdefendr Please see https://github.com/ggerganov/llama.cpp/issues/5807#issuecomment-1991179569 for a solution on how to convert Smaug-72B-v0.1 to GGUF.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.