wrong number of tensors for AdaptLLM/medicine-chat

[maziyarpanahi]

Using the convert script to convert this model AdaptLLM/medicine-chat to GGUF:

Set model parameters
gguf: context length = 4096
gguf: embedding length = 4096
gguf: feed forward length = 11008
gguf: head count = 32
gguf: key-value head count = 32
gguf: rms norm epsilon = 1e-05
gguf: file type = 1
Set model tokenizer
gguf: Setting special token type bos to 1
gguf: Setting special token type eos to 2
gguf: Setting special token type unk to 0
gguf: Setting special token type pad to 32000
gguf: Setting add_bos_token to True
gguf: Setting add_eos_token to False
Exporting model to 'quantized/AdaptLLM/medicine-chat/medicine-chat.fp16.gguf'
gguf: loading model part 'model-00001-of-00003.safetensors'
token_embd.weight, n_dims = 2, torch.float32 --> float16
blk.0.attn_norm.weight, n_dims = 1, torch.float32 --> float32
blk.0.ffn_down.weight, n_dims = 2, torch.float32 --> float16
blk.0.ffn_gate.weight, n_dims = 2, torch.float32 --> float16
blk.0.ffn_up.weight, n_dims = 2, torch.float32 --> float16
blk.0.ffn_norm.weight, n_dims = 1, torch.float32 --> float32
blk.0.attn_k.weight, n_dims = 2, torch.float32 --> float16
blk.0.attn_output.weight, n_dims = 2, torch.float32 --> float16
blk.0.attn_q.weight, n_dims = 2, torch.float32 --> float16
blk.0.attn_rot_embd, n_dims = 1, torch.float32 --> float32
blk.0.attn_v.weight, n_dims = 2, torch.float32 --> float16
blk.1.attn_norm.weight, n_dims = 1, torch.float32 --> float32
blk.1.ffn_down.weight, n_dims = 2, torch.float32 --> float16
blk.1.ffn_gate.weight, n_dims = 2, torch.float32 --> float16
blk.1.ffn_up.weight, n_dims = 2, torch.float32 --> float16
blk.1.ffn_norm.weight, n_dims = 1, torch.float32 --> float32
blk.1.attn_k.weight, n_dims = 2, torch.float32 --> float16
blk.1.attn_output.weight, n_dims = 2, torch.float32 --> float16
blk.1.attn_q.weight, n_dims = 2, torch.float32 --> float16
blk.1.attn_rot_embd, n_dims = 1, torch.float32 --> float32
blk.1.attn_v.weight, n_dims = 2, torch.float32 --> float16
blk.10.attn_norm.weight, n_dims = 1, torch.float32 --> float32
blk.10.ffn_down.weight, n_dims = 2, torch.float32 --> float16
blk.10.ffn_gate.weight, n_dims = 2, torch.float32 --> float16
blk.10.ffn_up.weight, n_dims = 2, torch.float32 --> float16
blk.10.ffn_norm.weight, n_dims = 1, torch.float32 --> float32
blk.10.attn_k.weight, n_dims = 2, torch.float32 --> float16
blk.10.attn_output.weight, n_dims = 2, torch.float32 --> float16
....
blk.28.attn_q.weight, n_dims = 2, torch.float32 --> float16
blk.28.attn_rot_embd, n_dims = 1, torch.float32 --> float32
blk.28.attn_v.weight, n_dims = 2, torch.float32 --> float16
blk.29.attn_norm.weight, n_dims = 1, torch.float32 --> float32
blk.29.ffn_down.weight, n_dims = 2, torch.float32 --> float16
blk.29.ffn_gate.weight, n_dims = 2, torch.float32 --> float16
blk.29.ffn_up.weight, n_dims = 2, torch.float32 --> float16
blk.29.ffn_norm.weight, n_dims = 1, torch.float32 --> float32
blk.29.attn_k.weight, n_dims = 2, torch.float32 --> float16
blk.29.attn_output.weight, n_dims = 2, torch.float32 --> float16
blk.29.attn_q.weight, n_dims = 2, torch.float32 --> float16
blk.29.attn_rot_embd, n_dims = 1, torch.float32 --> float32
blk.29.attn_v.weight, n_dims = 2, torch.float32 --> float16
blk.30.attn_norm.weight, n_dims = 1, torch.float32 --> float32
blk.30.ffn_down.weight, n_dims = 2, torch.float32 --> float16
blk.30.ffn_gate.weight, n_dims = 2, torch.float32 --> float16
blk.30.ffn_up.weight, n_dims = 2, torch.float32 --> float16
blk.30.ffn_norm.weight, n_dims = 1, torch.float32 --> float32
blk.30.attn_k.weight, n_dims = 2, torch.float32 --> float16
blk.30.attn_output.weight, n_dims = 2, torch.float32 --> float16
blk.30.attn_q.weight, n_dims = 2, torch.float32 --> float16
blk.30.attn_rot_embd, n_dims = 1, torch.float32 --> float32
blk.30.attn_v.weight, n_dims = 2, torch.float32 --> float16
blk.31.attn_norm.weight, n_dims = 1, torch.float32 --> float32
blk.31.ffn_down.weight, n_dims = 2, torch.float32 --> float16
blk.31.ffn_gate.weight, n_dims = 2, torch.float32 --> float16
blk.31.ffn_up.weight, n_dims = 2, torch.float32 --> float16
blk.31.ffn_norm.weight, n_dims = 1, torch.float32 --> float32
blk.31.attn_k.weight, n_dims = 2, torch.float32 --> float16
blk.31.attn_output.weight, n_dims = 2, torch.float32 --> float16
blk.31.attn_q.weight, n_dims = 2, torch.float32 --> float16
blk.31.attn_rot_embd, n_dims = 1, torch.float32 --> float32
blk.31.attn_v.weight, n_dims = 2, torch.float32 --> float16
output_norm.weight, n_dims = 1, torch.float32 --> float32

Inference:

main: build = 2602 (4bcd6b95)
main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
main: seed  = 1712249717
llama_model_loader: loaded meta data with 22 key-value pairs and 323 tensors from quantized/AdaptLLM/medicine-chat/medicine-chat.fp16.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = models--AdaptLLM--medicine-chat
llama_model_loader: - kv   2:                          llama.block_count u32              = 32
llama_model_loader: - kv   3:                       llama.context_length u32              = 4096
llama_model_loader: - kv   4:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 11008
llama_model_loader: - kv   6:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   7:              llama.attention.head_count_kv u32              = 32
llama_model_loader: - kv   8:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv   9:                          general.file_type u32              = 1
llama_model_loader: - kv  10:                           llama.vocab_size u32              = 32001
llama_model_loader: - kv  11:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  12:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,32001]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr[f32,32001]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr[i32,32001]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  19:            tokenizer.ggml.padding_token_id u32              = 32000
llama_model_loader: - kv  20:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  21:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - type  f32:   97 tensors
llama_model_loader: - type  f16:  226 tensors
llm_load_vocab: mismatch in special tokens definition ( 260/32001 vs 259/32001 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32001
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 4096
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 32
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 1
llm_load_print_meta: n_embd_k_gqa     = 4096
llm_load_print_meta: n_embd_v_gqa     = 4096
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 11008
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 4096
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = F16
llm_load_print_meta: model params     = 6.74 B
llm_load_print_meta: model size       = 12.55 GiB (16.00 BPW)
llm_load_print_meta: general.name     = models--AdaptLLM--medicine-chat
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: PAD token        = 32000 '<pad>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.12 MiB
llama_model_load: error loading model: done_getting_tensors: wrong number of tensors; expected 323, got 291
llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model 'quantized/AdaptLLM/medicine-chat/medicine-chat.fp16.gguf'
main: error: unable to load model

[eleius]

Also getting "error loading model: done_getting_tensors: wrong number of tensors; expected 323, got 291"

[MoonRide303]

Same problem with tulu-2-13b converted with convert-hf-to-gguf.py, then Q6_K via quantize:

llama_model_load: error loading model: done_getting_tensors: wrong number of tensors; expected 403, got 363
llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model '.\Tulu-2-13B-Q6_K.gguf'
main: error: unable to load model

Q6_K GGUF from https://huggingface.co/TheBloke/tulu-2-13B-GGUF (created with older version of llama.cpp) worked fine.

[maziyarpanahi]

So something must have changed to cause this. That's a good start, thanks @MoonRide303

[xxxxyu]

Also getting this when converting Llama-2-7b-hf from huggingface with convert-hf-to-gguf.py

[xxxxyu]

Update

I just tried with a local clone of https://huggingface.co/meta-llama/Llama-2-7b/tree/main, and the converted model (via convert.py) works fine. I think this is a more flexible walkaround for now, rather than manually downloading each quantized version.

It seems @MoonRide303 is also using convert-hf-to-gguf.py. So I guess there is something to do with the huggingface pipeline and convert-hf-to-gguf.py?

[xudong163]

I also enter this problem , using convert-hf-to-gguf.py， Any one has some solution?

[MoonRide303]

@xxxxyu I use convert-hf-to-gguf.py with locally downloaded HF repos.

[cotwitch]

Getting the same error while using model Czech-GPT-2-XL-133k-GGUF.

llama.cpp error: 'done_getting_tensors: wrong number of tensors; expected 582, got 581'

[slaren]

@cotwitch If you look at that model with gguf-dump.py, you will see that it has the tensor output.weight duplicated. Not sure how that happened, but that's not a valid model.

[maziyarpanahi]

@slaren interesting! do you think this can be a result of some sort of merging?

[slaren]

Sorry, I do not have any insights about how that may have happened. I guess it is a bug in the conversion script, and the gguf-py library should have prevented it from happening, but I don't know the cause.

[ffflowww]

qucik fix for me was to replace those line

# we don't need these
if name.endswith((".attention.masked_bias", ".attention.bias", ".attention.rotary_emb.inv_freq")):
    continue

with:

# we don't need these
if name.endswith((".attention.masked_bias", ".attention.bias", ".attention.rotary_emb.inv_freq", "self_attn.rotary_emb.inv_freq")):
    continue

basically just adding extra "self_attn.rotary_emb.inv_freq" for skipping in convert-hf-to-gguf.py

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.