GPTQ-for-LLaMa
GPTQ-for-LLaMa copied to clipboard
qwopqwop200
Having trouble using saved models
When I try and use the model, I see errors on very layer in the model:
size mismatch for model.layers.77.mlp.down_proj.scales: copying a param with shape torch.Size([8192, 1]) from checkpoint, the shape in current model is torch.Size([172, 8192]).
size mismatch for model.layers.77.mlp.up_proj.scales: copying a param with shape torch.Size([22016, 1]) from checkpoint, the shape in current model is torch.Size([64, 22016]).
size mismatch for model.layers.78.self_attn.q_proj.scales: copying a param with shape torch.Size([8192, 1]) from checkpoint, the shape in current model is torch.Size([64, 8192]).
size mismatch for model.layers.78.self_attn.k_proj.scales: copying a param with shape torch.Size([8192, 1]) from checkpoint, the shape in current model is torch.Size([64, 8192]).
size mismatch for model.layers.78.self_attn.v_proj.scales: copying a param with shape torch.Size([8192, 1]) from checkpoint, the shape in current model is torch.Size([64, 8192]).
size mismatch for model.layers.78.self_attn.o_proj.scales: copying a param with shape torch.Size([8192, 1]) from checkpoint, the shape in current model is torch.Size([64, 8192]).
size mismatch for model.layers.78.mlp.gate_proj.scales: copying a param with shape torch.Size([22016, 1]) from checkpoint, the shape in current model is torch.Size([64, 22016]).
size mismatch for model.layers.78.mlp.down_proj.scales: copying a param with shape torch.Size([8192, 1]) from checkpoint, the shape in current model is torch.Size([172, 8192]).
size mismatch for model.layers.78.mlp.up_proj.scales: copying a param with shape torch.Size([22016, 1]) from checkpoint, the shape in current model is torch.Size([64, 22016]).
size mismatch for model.layers.79.self_attn.q_proj.scales: copying a param with shape torch.Size([8192, 1]) from checkpoint, the shape in current model is torch.Size([64, 8192]).
size mismatch for model.layers.79.self_attn.k_proj.scales: copying a param with shape torch.Size([8192, 1]) from checkpoint, the shape in current model is torch.Size([64, 8192]).
size mismatch for model.layers.79.self_attn.v_proj.scales: copying a param with shape torch.Size([8192, 1]) from checkpoint, the shape in current model is torch.Size([64, 8192]).
size mismatch for model.layers.79.self_attn.o_proj.scales: copying a param with shape torch.Size([8192, 1]) from checkpoint, the shape in current model is torch.Size([64, 8192]).
size mismatch for model.layers.79.mlp.gate_proj.scales: copying a param with shape torch.Size([22016, 1]) from checkpoint, the shape in current model is torch.Size([64, 22016]).
size mismatch for model.layers.79.mlp.down_proj.scales: copying a param with shape torch.Size([8192, 1]) from checkpoint, the shape in current model is torch.Size([172, 8192]).
size mismatch for model.layers.79.mlp.up_proj.scales: copying a param with shape torch.Size([22016, 1]) from checkpoint, the shape in current model is torch.Size([64, 22016]).
I and using this commend to run the model: python llama.py /mnt/data2/llama_weighs/llama-65b-hf/ c4 --wbits 4 --groupsize 128 --load /mnt/data2/llama_weighs/llama-65b-hf/llama65b-4bit.pt --benchmark 2048 --check
I moved the generated model "llama65b-4bit.pt" into the Hugging Face model directory. Not sure if this was the right thing to do...
+1 Hmm yes i also had this problem when trying to save a alpaca lora checkpoint into ggl and then quantize it the - -check command fails.
+1 I am facing a similar issue when loading from here https://huggingface.co/maderix/llama-65b-4bit. Any help on this would be great !
The code has changed due to current grouping support. please quantization the LLaMa model once again.
@qwopqwop200 I am seeing this error with T5 based models on the current T5 branch after quantization procedure.
Tried both latest HF transformers and your latest transformers fork.
example for t5-flan-small: size mismatch for decoder.block.7.layer.2.DenseReluDense.wo.scales: copying a param with shape torch.Size([8, 512]) from checkpoint, the shape in current model is torch.Size([1, 512]).
