Size mismatch

[anujnayyar1]

Hey! First thanks so so much for releasing your awesome work!!!

Iv'e tried to export my fine-tuned OPT-13B model using the export_int8_model.py

When then trying to load this model in Int8OPTForCausalLM.from_pretrained() I run into the errors below.

The model was fine-tuned with a block size of 2048 (using the HF clm example script) I have tried re-exporting in int8, using the seq length of 2048 and using 'ignore_mismatched_sizes=True' but still no luck

I'd really appreciate any insights here to get this working!

raise RuntimeError(f"Error(s) in loading state_dict for {model.__class__.__name__}:\n\t{error_msg}") RuntimeError: Error(s) in loading state_dict for Int8OPTForCausalLM: size mismatch for model.decoder.layers.0.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.0.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.0.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.0.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.0.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint,the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.0.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.1.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.1.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.1.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.1.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.1.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint,the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.1.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.2.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.2.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.2.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.2.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.2.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint,the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.2.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.3.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.3.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.3.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.3.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.3.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint,the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.3.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.4.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.4.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.4.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.4.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.4.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint,the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.4.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.5.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.5.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.5.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.5.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.5.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint,the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.5.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.6.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.6.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.6.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.6.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.6.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint,the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.6.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.7.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.7.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.7.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.7.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.7.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint,the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.7.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.8.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.8.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.8.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.8.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.8.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint,the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.8.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.9.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.9.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.9.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) fromcheckpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.9.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.9.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint,the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.9.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.10.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.10.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.10.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.10.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.10.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.10.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.11.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.11.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.11.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.11.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.11.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.11.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.12.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.12.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.12.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.12.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.12.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.12.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.13.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.13.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.13.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.13.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.13.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.13.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.14.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.14.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.14.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.14.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.14.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.14.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.15.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.15.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.15.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.15.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.15.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.15.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.16.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.16.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.16.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.16.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.16.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.16.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.17.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.17.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.17.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.17.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.17.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.17.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.18.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.18.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.18.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.18.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.18.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.18.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.19.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.19.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.19.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.19.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.19.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.19.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.20.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.20.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.20.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.20.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.20.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.20.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.21.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.21.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.21.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.21.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.21.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.21.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.22.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.22.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.22.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.22.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.22.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.22.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.23.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.23.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.23.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.23.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.23.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.23.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.24.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.24.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.24.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.24.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.24.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.24.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.25.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.25.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.25.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.25.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.25.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.25.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.26.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.26.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.26.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.26.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.26.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.26.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.27.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.27.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.27.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.27.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.27.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.27.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.28.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.28.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.28.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.28.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.28.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.28.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.29.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.29.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.29.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.29.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.29.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.29.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.30.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.30.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.30.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.30.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.30.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.30.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.31.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.31.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.31.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.31.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.31.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.31.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.32.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.32.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.32.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.32.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.32.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.32.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.33.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.33.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.33.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.33.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.33.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.33.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.34.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.34.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.34.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.34.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.34.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.34.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.35.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.35.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.35.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.35.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.35.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.35.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.36.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.36.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.36.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.36.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.36.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.36.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.37.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.37.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.37.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.37.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.37.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.37.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.38.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.38.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.38.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.38.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.38.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.38.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.39.self_attn.k_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.39.self_attn.v_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.39.self_attn.q_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.39.self_attn.out_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1, 5120]). size mismatch for model.decoder.layers.39.fc1.bias: copying a param with shape torch.Size([20480]) from checkpoint, the shape in current model is torch.Size([1, 20480]). size mismatch for model.decoder.layers.39.fc2.bias: copying a param with shape torch.Size([5120]) from checkpoint,the shape in current model is torch.Size([1, 5120]). You may consider adding ignore_mismatched_sizes=Truein the modelfrom_pretrained method.

[imargulis]

@anujnayyar1, you can easily fix this issue by updating from_float methods of the corresponding layers, implemented in torch-int package. For example for W8A8B8O8Linear:

        int8_module.weight = torch.reshape(int8_weight,int8_module.weight.shape)
        int8_module.bias = torch.reshape(int8_bias, int8_module.bias.shape)

ensures there is no size mismatch.