resulting checkpoint can be compatible with clip.load()?

[KyonP]

This repo greatly helped my finetuning process. Thanks.

I am wondering if these resulting checkpoints, which are produced by this code in lightning_logs/version_#/checkpoints/~~~.ckpt, are compatible with clip.load?

I could torch.load the checkpoints, but not clip.load, giving me the following error:

>>> clip.load('save/checkpoints/vit-b-32_fine.ckpt')

PyDev console: starting.
Traceback (most recent call last):
  File "/root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
  File "/root/anaconda3/envs/ldm/lib/python3.8/site-packages/clip/clip.py", line 136, in load
    state_dict = torch.load(opened_file, map_location="cpu")
  File "/root/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/serialization.py", line 713, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/root/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/serialization.py", line 920, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
EOFError: Ran out of input

Is there another way to load the checkpoint to pretrained CLIP model (ViT-B/32)?

[KyonP]

Also, I am getting missing key in state_dict error from loading the checkpoint using torch.load_state_dict method.

     from mycode.clip_finetune.models import wrapper

            clp, preprocess = clip.load(clip_model)

            for p in clp.parameters():
                p.requires_grad = False

            model = wrapper.CustomCLIPWrapper(clp.visual, clp.transformer, batch_size, avg_word_embs=True)

            model.model.token_embedding = clp.token_embedding
            model.model.ln_final = clp.ln_final
            model.model.text_projection = clp.text_projection
            model.teacher = copy.deepcopy(model.model)

            model.load_state_dict(torch.load('save/checkpoints/vit-b-32_fine.ckpt'))

And I have the following missing keys error.

Traceback (most recent call last):
  File "/root/.pycharm_helpers/pydev/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
  File "/root/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1604, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for CustomCLIPWrapper:
	Missing key(s) in state_dict: "sink_temp", "model.positional_embedding", "model.text_projection", "model.logit_scale", "model.token_embedding.weight", "model.ln_final.weight", "model.ln_final.bias", "model.visual.class_embedding", "model.visual.positional_embedding", "model.visual.proj", "model.visual.conv1.weight", "model.visual.ln_pre.weight", "model.visual.ln_pre.bias", "model.visual.transformer.resblocks.0.attn.in_proj_weight", "model.visual.transformer.resblocks.0.attn.in_proj_bias", "model.visual.transformer.resblocks.0.attn.out_proj.weight", "model.visual.transformer.resblocks.0.attn.out_proj.bias", "model.visual.transformer.resblocks.0.ln_1.weight", "model.visual.transformer.resblocks.0.ln_1.bias", "model.visual.transformer.resblocks.0.mlp.c_fc.weight", "model.visual.transformer.resblocks.0.mlp.c_fc.bias", "model.visual.transformer.resblocks.0.mlp.c_proj.weight", "model.visual.transformer.resblocks.0.mlp.c_proj.bias", "model.visual.transformer.resblocks.0.ln_2.weight", "model.visual.transformer.resblocks.0.ln_2.bias", "model.visual.transformer.resblocks.1.attn.in_proj_weight", "model.visual.transformer.resblocks.1.attn.in_proj_bias", "model.visual.transformer.resblocks.1.attn.out_proj.weight", "model.visual.transformer.resblocks.1.attn.out_proj.bias", "model.visual.transformer.resblocks.1.ln_1.weight", "model.visual.transformer.resblocks.1.ln_1.bias", "model.visual.transformer.resblocks.1.mlp.c_fc.weight", "model.visual.transformer.resblocks.1.mlp.c_fc.bias", "model.visual.transformer.resblocks.1.mlp.c_proj.weight", "model.visual.transformer.resblocks.1.mlp.c_proj.bias", "model.visual.transformer.resblocks.1.ln_2.weight", "model.visual.transformer.resblocks.1.ln_2.bias", "model.visual.transformer.resblocks.2.attn.in_proj_weight", "model.visual.transformer.resblocks.2.attn.in_proj_bias", "model.visual.transformer.resblocks.2.attn.out_proj.weight", "model.visual.transformer.resblocks.2.attn.out_proj.bias", "model.visual.transformer.resblocks.2.ln_1.weight", "model.visual.transformer.resblocks.2.ln_1.bias", "model.visual.transformer.resblocks.2.mlp.c_fc.weight", "model.visual.transformer.resblocks.2.mlp.c_fc.bias", "model.visual.transformer.resblocks.2.mlp.c_proj.weight", "model.visual.transformer.resblocks.2.mlp.c_proj.bias", "model.visual.transformer.resblocks.2.ln_2.weight", "model.visual.transformer.resblocks.2.ln_2.bias", "model.visual.transformer.resblocks.3.attn.in_proj_weight", "model.visual.transformer.resblocks.3.attn.in_proj_bias", "model.visual.transformer.resblocks.3.attn.out_proj.weight", "model.visual.transformer.resblocks.3.attn.out_proj.bias", "model.visual.transformer.resblocks.3.ln_1.weight", "model.visual.transformer.resblocks.3.ln_1.bias", "model.visual.transformer.resblocks.3.mlp.c_fc.weight", "model.visual.transformer.resblocks.3.mlp.c_fc.bias", "model.visual.transformer.resblocks.3.mlp.c_proj.weight", "model.visual.transformer.resblocks.3.mlp.c_proj.bias", "model.visual.transformer.resblocks.3.ln_2.weight", "model.visual.transformer.resblocks.3.ln_2.bias", "model.visual.transformer.resblocks.4.attn.in_proj_weight", "model.visual.transformer.resblocks.4.attn.in_proj_bias", "model.visual.transformer.resblocks.4.attn.out_proj.weight", "model.visual.transformer.resblocks.4.attn.out_proj.bias", "model.visual.transformer.resblocks.4.ln_1.weight", "model.visual.transformer.resblocks.4.ln_1.bias", "model.visual.transformer.resblocks.4.mlp.c_fc.weight", "model.visual.transformer.resblocks.4.mlp.c_fc.bias", "model.visual.transformer.resblocks.4.mlp.c_proj.weight", "model.visual.transformer.resblocks.4.mlp.c_proj.bias", "model.visual.transformer.resblocks.4.ln_2.weight", "model.visual.transformer.resblocks.4.ln_2.bias", "model.visual.transformer.resblocks.5.attn.in_proj_weight", "model.visual.transformer.resblocks.5.attn.in_proj_bias", "model.visual.transformer.resblocks.5.attn.out_proj.weight", "model.visual.transformer.resblocks.5.attn.out_proj.bias", "model.visual.transformer.resblocks.5.ln_1.weight", "model.visual.transformer.resblocks.5.ln_1.bias", "model.visual.transformer.resblocks.5.mlp.c_fc.weight", "model.visual.transformer.resblocks.5.mlp.c_fc.bias", "model.visual.transformer.resblocks.5.mlp.c_proj.weight", "model.visual.transformer.resblocks.5.mlp.c_proj.bias", "model.visual.transformer.resblocks.5.ln_2.weight", "model.visual.transformer.resblocks.5.ln_2.bias", "model.visual.transformer.resblocks.6.attn.in_proj_weight", "model.visual.transformer.resblocks.6.attn.in_proj_bias", "model.visual.transformer.resblocks.6.attn.out_proj.weight", "model.visual.transformer.resblocks.6.attn.out_proj.bias", "model.visual.transformer.resblocks.6.ln_1.weight", "model.visual.transformer.resblocks.6.ln_1.bias", "model.visual.transformer.resblocks.6.mlp.c_fc.weight", "model.visual.transformer.resblocks.6.mlp.c_fc.bias", "model.visual.transformer.resblocks.6.mlp.c_proj.weight", "model.visual.transformer.resblocks.6.mlp.c_proj.bias", "model.visual.transformer.resblocks.6.ln_2.weight", "model.visual.transformer.resblocks.6.ln_2.bias", "model.visual.transformer.resblocks.7.attn.in_proj_weight", "model.visual.transformer.resblocks.7.attn.in_proj_bias", "model.visual.transformer.resblocks.7.attn.out_proj.weight", "model.visual.transformer.resblocks.7.attn.out_proj.bias", "model.visual.transformer.resblocks.7.ln_1.weight", "model.visual.transformer.resblocks.7.ln_1.bias", "model.visual.transformer.resblocks.7.mlp.c_fc.weight", "model.visual.transformer.resblocks.7.mlp.c_fc.bias", "model.visual.transformer.resblocks.7.mlp.c_proj.weight", "model.visual.transformer.resblocks.7.mlp.c_proj.bias", "model.visual.transformer.resblocks.7.ln_2.weight", "model.visual.transformer.resblocks.7.ln_2.bias", "model.visual.transformer.resblocks.8.attn.in_proj_weight", "model.visual.transformer.resblocks.8.attn.in_proj_bias", "model.visual.transformer.resblocks.8.attn.out_proj.weight", "model.visual.transformer.resblocks.8.attn.out_proj.bias", "model.visual.transformer.resblocks.8.ln_1.weight", "model.visual.transformer.resblocks.8.ln_1.bias", "model.visual.transformer.resblocks.8.mlp.c_fc.weight", "model.visual.transformer.resblocks.8.mlp.c_fc.bias", "model.visual.transformer.resblocks.8.mlp.c_proj.weight", "model.visual.transformer.resblocks.8.mlp.c_proj.bias", "model.visual.transformer.resblocks.8.ln_2.weight", "model.visual.transformer.resblocks.8.ln_2.bias", "model.visual.transformer.resblocks.9.attn.in_proj_weight", "model.visual.transformer.resblocks.9.attn.in_proj_bias", "model.visual.transformer.resblocks.9.attn.out_proj.weight", "model.visual.transformer.resblocks.9.attn.out_proj.bias", "model.visual.transformer.resblocks.9.ln_1.weight", "model.visual.transformer.resblocks.9.ln_1.bias", "model.visual.transformer.resblocks.9.mlp.c_fc.weight", "model.visual.transformer.resblocks.9.mlp.c_fc.bias", "model.visual.transformer.resblocks.9.mlp.c_proj.weight", "model.visual.transformer.resblocks.9.mlp.c_proj.bias", "model.visual.transformer.resblocks.9.ln_2.weight", "model.visual.transformer.resblocks.9.ln_2.bias", "model.visual.transformer.resblocks.10.attn.in_proj_weight", "model.visual.transformer.resblocks.10.attn.in_proj_bias", "model.visual.transformer.resblocks.10.attn.out_proj.weight", "model.visual.transformer.resblocks.10.attn.out_proj.bias", "model.visual.transformer.resblocks.10.ln_1.weight", "model.visual.transformer.resblocks.10.ln_1.bias", "model.visual.transformer.resblocks.10.mlp.c_fc.weight", "model.visual.transformer.resblocks.10.mlp.c_fc.bias", "model.visual.transformer.resblocks.10.mlp.c_proj.weight", "model.visual.transformer.resblocks.10.mlp.c_proj.bias", "model.visual.transformer.resblocks.10.ln_2.weight", "model.visual.transformer.resblocks.10.ln_2.bias", "model.visual.transformer.resblocks.11.attn.in_proj_weight", "model.visual.transformer.resblocks.11.attn.in_proj_bias", "model.visual.transformer.resblocks.11.attn.out_proj.weight", "model.visual.transformer.resblocks.11.attn.out_proj.bias", "model.visual.transformer.resblocks.11.ln_1.weight", "model.visual.transformer.resblocks.11.ln_1.bias", "model.visual.transformer.resblocks.11.mlp.c_fc.weight", "model.visual.transformer.resblocks.11.mlp.c_fc.bias", "model.visual.transformer.resblocks.11.mlp.c_proj.weight", "model.visual.transformer.resblocks.11.mlp.c_proj.bias", "model.visual.transformer.resblocks.11.ln_2.weight", "model.visual.transformer.resblocks.11.ln_2.bias", "model.visual.ln_post.weight", "model.visual.ln_post.bias", "model.transformer.resblocks.0.attn.in_proj_weight", "model.transformer.resblocks.0.attn.in_proj_bias", "model.transformer.resblocks.0.attn.out_proj.weight", "model.transformer.resblocks.0.attn.out_proj.bias", "model.transformer.resblocks.0.ln_1.weight", "model.transformer.resblocks.0.ln_1.bias", "model.transformer.resblocks.0.mlp.c_fc.weight", "model.transformer.resblocks.0.mlp.c_fc.bias", "model.transformer.resblocks.0.mlp.c_proj.weight", "model.transformer.resblocks.0.mlp.c_proj.bias", "model.transformer.resblocks.0.ln_2.weight", "model.transformer.resblocks.0.ln_2.bias", "model.transformer.resblocks.1.attn.in_proj_weight", "model.transformer.resblocks.1.attn.in_proj_bias", "model.transformer.resblocks.1.attn.out_proj.weight", "model.transformer.resblocks.1.attn.out_proj.bias", "model.transformer.resblocks.1.ln_1.weight", "model.transformer.resblocks.1.ln_1.bias", "model.transformer.resblocks.1.mlp.c_fc.weight", "model.transformer.resblocks.1.mlp.c_fc.bias", "model.transformer.resblocks.1.mlp.c_proj.weight", "model.transformer.resblocks.1.mlp.c_proj.bias", "model.transformer.resblocks.1.ln_2.weight", "model.transformer.resblocks.1.ln_2.bias", "model.transformer.resblocks.2.attn.in_proj_weight", "model.transformer.resblocks.2.attn.in_proj_bias", "model.transformer.resblocks.2.attn.out_proj.weight", "model.transformer.resblocks.2.attn.out_proj.bias", "model.transformer.resblocks.2.ln_1.weight", "model.transformer.resblocks.2.ln_1.bias", "model.transformer.resblocks.2.mlp.c_fc.weight", "model.transformer.resblocks.2.mlp.c_fc.bias", "model.transformer.resblocks.2.mlp.c_proj.weight", "model.transformer.resblocks.2.mlp.c_proj.bias", "model.transformer.resblocks.2.ln_2.weight", "model.transformer.resblocks.2.ln_2.bias", "model.transformer.resblocks.3.attn.in_proj_weight", "model.transformer.resblocks.3.attn.in_proj_bias", "model.transformer.resblocks.3.attn.out_proj.weight", "model.transformer.resblocks.3.attn.out_proj.bias", "model.transformer.resblocks.3.ln_1.weight", "model.transformer.resblocks.3.ln_1.bias", "model.transformer.resblocks.3.mlp.c_fc.weight", "model.transformer.resblocks.3.mlp.c_fc.bias", "model.transformer.resblocks.3.mlp.c_proj.weight", "model.transformer.resblocks.3.mlp.c_proj.bias", "model.transformer.resblocks.3.ln_2.weight", "model.transformer.resblocks.3.ln_2.bias", "model.transformer.resblocks.4.attn.in_proj_weight", "model.transformer.resblocks.4.attn.in_proj_bias", "model.transformer.resblocks.4.attn.out_proj.weight", "model.transformer.resblocks.4.attn.out_proj.bias", "model.transformer.resblocks.4.ln_1.weight", "model.transformer.resblocks.4.ln_1.bias", "model.transformer.resblocks.4.mlp.c_fc.weight", "model.transformer.resblocks.4.mlp.c_fc.bias", "model.transformer.resblocks.4.mlp.c_proj.weight", "model.transformer.resblocks.4.mlp.c_proj.bias", "model.transformer.resblocks.4.ln_2.weight", "model.transformer.resblocks.4.ln_2.bias", "model.transformer.resblocks.5.attn.in_proj_weight", "model.transformer.resblocks.5.attn.in_proj_bias", "model.transformer.resblocks.5.attn.out_proj.weight", "model.transformer.resblocks.5.attn.out_proj.bias", "model.transformer.resblocks.5.ln_1.weight", "model.transformer.resblocks.5.ln_1.bias", "model.transformer.resblocks.5.mlp.c_fc.weight", "model.transformer.resblocks.5.mlp.c_fc.bias", "model.transformer.resblocks.5.mlp.c_proj.weight", "model.transformer.resblocks.5.mlp.c_proj.bias", "model.transformer.resblocks.5.ln_2.weight", "model.transformer.resblocks.5.ln_2.bias", "model.transformer.resblocks.6.attn.in_proj_weight", "model.transformer.resblocks.6.attn.in_proj_bias", "model.transformer.resblocks.6.attn.out_proj.weight", "model.transformer.resblocks.6.attn.out_proj.bias", "model.transformer.resblocks.6.ln_1.weight", "model.transformer.resblocks.6.ln_1.bias", "model.transformer.resblocks.6.mlp.c_fc.weight", "model.transformer.resblocks.6.mlp.c_fc.bias", "model.transformer.resblocks.6.mlp.c_proj.weight", "model.transformer.resblocks.6.mlp.c_proj.bias", "model.transformer.resblocks.6.ln_2.weight", "model.transformer.resblocks.6.ln_2.bias", "model.transformer.resblocks.7.attn.in_proj_weight", "model.transformer.resblocks.7.attn.in_proj_bias", "model.transformer.resblocks.7.attn.out_proj.weight", "model.transformer.resblocks.7.attn.out_proj.bias", "model.transformer.resblocks.7.ln_1.weight", "model.transformer.resblocks.7.ln_1.bias", "model.transformer.resblocks.7.mlp.c_fc.weight", "model.transformer.resblocks.7.mlp.c_fc.bias", "model.transformer.resblocks.7.mlp.c_proj.weight", "model.transformer.resblocks.7.mlp.c_proj.bias", "model.transformer.resblocks.7.ln_2.weight", "model.transformer.resblocks.7.ln_2.bias", "model.transformer.resblocks.8.attn.in_proj_weight", "model.transformer.resblocks.8.attn.in_proj_bias", "model.transformer.resblocks.8.attn.out_proj.weight", "model.transformer.resblocks.8.attn.out_proj.bias", "model.transformer.resblocks.8.ln_1.weight", "model.transformer.resblocks.8.ln_1.bias", "model.transformer.resblocks.8.mlp.c_fc.weight", "model.transformer.resblocks.8.mlp.c_fc.bias", "model.transformer.resblocks.8.mlp.c_proj.weight", "model.transformer.resblocks.8.mlp.c_proj.bias", "model.transformer.resblocks.8.ln_2.weight", "model.transformer.resblocks.8.ln_2.bias", "model.transformer.resblocks.9.attn.in_proj_weight", "model.transformer.resblocks.9.attn.in_proj_bias", "model.transformer.resblocks.9.attn.out_proj.weight", "model.transformer.resblocks.9.attn.out_proj.bias", "model.transformer.resblocks.9.ln_1.weight", "model.transformer.resblocks.9.ln_1.bias", "model.transformer.resblocks.9.mlp.c_fc.weight", "model.transformer.resblocks.9.mlp.c_fc.bias", "model.transformer.resblocks.9.mlp.c_proj.weight", "model.transformer.resblocks.9.mlp.c_proj.bias", "model.transformer.resblocks.9.ln_2.weight", "model.transformer.resblocks.9.ln_2.bias", "model.transformer.resblocks.10.attn.in_proj_weight", "model.transformer.resblocks.10.attn.in_proj_bias", "model.transformer.resblocks.10.attn.out_proj.weight", "model.transformer.resblocks.10.attn.out_proj.bias", "model.transformer.resblocks.10.ln_1.weight", "model.transformer.resblocks.10.ln_1.bias", "model.transformer.resblocks.10.mlp.c_fc.weight", "model.transformer.resblocks.10.mlp.c_fc.bias", "model.transformer.resblocks.10.mlp.c_proj.weight", "model.transformer.resblocks.10.mlp.c_proj.bias", "model.transformer.resblocks.10.ln_2.weight", "model.transformer.resblocks.10.ln_2.bias", "model.transformer.resblocks.11.attn.in_proj_weight", "model.transformer.resblocks.11.attn.in_proj_bias", "model.transformer.resblocks.11.attn.out_proj.weight", "model.transformer.resblocks.11.attn.out_proj.bias", "model.transformer.resblocks.11.ln_1.weight", "model.transformer.resblocks.11.ln_1.bias", "model.transformer.resblocks.11.mlp.c_fc.weight", "model.transformer.resblocks.11.mlp.c_fc.bias", "model.transformer.resblocks.11.mlp.c_proj.weight", "model.transformer.resblocks.11.mlp.c_proj.bias", "model.transformer.resblocks.11.ln_2.weight", "model.transformer.resblocks.11.ln_2.bias", "teacher.positional_embedding", "teacher.text_projection", "teacher.logit_scale", "teacher.token_embedding.weight", "teacher.ln_final.weight", "teacher.ln_final.bias", "teacher.visual.class_embedding", "teacher.visual.positional_embedding", "teacher.visual.proj", "teacher.visual.conv1.weight", "teacher.visual.ln_pre.weight", "teacher.visual.ln_pre.bias", "teacher.visual.transformer.resblocks.0.attn.in_proj_weight", "teacher.visual.transformer.resblocks.0.attn.in_proj_bias", "teacher.visual.transformer.resblocks.0.attn.out_proj.weight", "teacher.visual.transformer.resblocks.0.attn.out_proj.bias", "teacher.visual.transformer.resblocks.0.ln_1.weight", "teacher.visual.transformer.resblocks.0.ln_1.bias", "teacher.visual.transformer.resblocks.0.mlp.c_fc.weight", "teacher.visual.transformer.resblocks.0.mlp.c_fc.bias", "teacher.visual.transformer.resblocks.0.mlp.c_proj.weight", "teacher.visual.transformer.resblocks.0.mlp.c_proj.bias", "teacher.visual.transformer.resblocks.0.ln_2.weight", "teacher.visual.transformer.resblocks.0.ln_2.bias", "teacher.visual.transformer.resblocks.1.attn.in_proj_weight", "teacher.visual.transformer.resblocks.1.attn.in_proj_bias", "teacher.visual.transformer.resblocks.1.attn.out_proj.weight", "teacher.visual.transformer.resblocks.1.attn.out_proj.bias", "teacher.visual.transformer.resblocks.1.ln_1.weight", "teacher.visual.transformer.resblocks.1.ln_1.bias", "teacher.visual.transformer.resblocks.1.mlp.c_fc.weight", "teacher.visual.transformer.resblocks.1.mlp.c_fc.bias", "teacher.visual.transformer.resblocks.1.mlp.c_proj.weight", "teacher.visual.transformer.resblocks.1.mlp.c_proj.bias", "teacher.visual.transformer.resblocks.1.ln_2.weight", "teacher.visual.transformer.resblocks.1.ln_2.bias", "teacher.visual.transformer.resblocks.2.attn.in_proj_weight", "teacher.visual.transformer.resblocks.2.attn.in_proj_bias", "teacher.visual.transformer.resblocks.2.attn.out_proj.weight", "teacher.visual.transformer.resblocks.2.attn.out_proj.bias", "teacher.visual.transformer.resblocks.2.ln_1.weight", "teacher.visual.transformer.resblocks.2.ln_1.bias", "teacher.visual.transformer.resblocks.2.mlp.c_fc.weight", "teacher.visual.transformer.resblocks.2.mlp.c_fc.bias", "teacher.visual.transformer.resblocks.2.mlp.c_proj.weight", "teacher.visual.transformer.resblocks.2.mlp.c_proj.bias", "teacher.visual.transformer.resblocks.2.ln_2.weight", "teacher.visual.transformer.resblocks.2.ln_2.bias", "teacher.visual.transformer.resblocks.3.attn.in_proj_weight", "teacher.visual.transformer.resblocks.3.attn.in_proj_bias", "teacher.visual.transformer.resblocks.3.attn.out_proj.weight", "teacher.visual.transformer.resblocks.3.attn.out_proj.bias", "teacher.visual.transformer.resblocks.3.ln_1.weight", "teacher.visual.transformer.resblocks.3.ln_1.bias", "teacher.visual.transformer.resblocks.3.mlp.c_fc.weight", "teacher.visual.transformer.resblocks.3.mlp.c_fc.bias", "teacher.visual.transformer.resblocks.3.mlp.c_proj.weight", "teacher.visual.transformer.resblocks.3.mlp.c_proj.bias", "teacher.visual.transformer.resblocks.3.ln_2.weight", "teacher.visual.transformer.resblocks.3.ln_2.bias", "teacher.visual.transformer.resblocks.4.attn.in_proj_weight", "teacher.visual.transformer.resblocks.4.attn.in_proj_bias", "teacher.visual.transformer.resblocks.4.attn.out_proj.weight", "teacher.visual.transformer.resblocks.4.attn.out_proj.bias", "teacher.visual.transformer.resblocks.4.ln_1.weight", "teacher.visual.transformer.resblocks.4.ln_1.bias", "teacher.visual.transformer.resblocks.4.mlp.c_fc.weight", "teacher.visual.transformer.resblocks.4.mlp.c_fc.bias", "teacher.visual.transformer.resblocks.4.mlp.c_proj.weight", "teacher.visual.transformer.resblocks.4.mlp.c_proj.bias", "teacher.visual.transformer.resblocks.4.ln_2.weight", "teacher.visual.transformer.resblocks.4.ln_2.bias", "teacher.visual.transformer.resblocks.5.attn.in_proj_weight", "teacher.visual.transformer.resblocks.5.attn.in_proj_bias", "teacher.visual.transformer.resblocks.5.attn.out_proj.weight", "teacher.visual.transformer.resblocks.5.attn.out_proj.bias", "teacher.visual.transformer.resblocks.5.ln_1.weight", "teacher.visual.transformer.resblocks.5.ln_1.bias", "teacher.visual.transformer.resblocks.5.mlp.c_fc.weight", "teacher.visual.transformer.resblocks.5.mlp.c_fc.bias", "teacher.visual.transformer.resblocks.5.mlp.c_proj.weight", "teacher.visual.transformer.resblocks.5.mlp.c_proj.bias", "teacher.visual.transformer.resblocks.5.ln_2.weight", "teacher.visual.transformer.resblocks.5.ln_2.bias", "teacher.visual.transformer.resblocks.6.attn.in_proj_weight", "teacher.visual.transformer.resblocks.6.attn.in_proj_bias", "teacher.visual.transformer.resblocks.6.attn.out_proj.weight", "teacher.visual.transformer.resblocks.6.attn.out_proj.bias", "teacher.visual.transformer.resblocks.6.ln_1.weight", "teacher.visual.transformer.resblocks.6.ln_1.bias", "teacher.visual.transformer.resblocks.6.mlp.c_fc.weight", "teacher.visual.transformer.resblocks.6.mlp.c_fc.bias", "teacher.visual.transformer.resblocks.6.mlp.c_proj.weight", "teacher.visual.transformer.resblocks.6.mlp.c_proj.bias", "teacher.visual.transformer.resblocks.6.ln_2.weight", "teacher.visual.transformer.resblocks.6.ln_2.bias", "teacher.visual.transformer.resblocks.7.attn.in_proj_weight", "teacher.visual.transformer.resblocks.7.attn.in_proj_bias", "teacher.visual.transformer.resblocks.7.attn.out_proj.weight", "teacher.visual.transformer.resblocks.7.attn.out_proj.bias", "teacher.visual.transformer.resblocks.7.ln_1.weight", "teacher.visual.transformer.resblocks.7.ln_1.bias", "teacher.visual.transformer.resblocks.7.mlp.c_fc.weight", "teacher.visual.transformer.resblocks.7.mlp.c_fc.bias", "teacher.visual.transformer.resblocks.7.mlp.c_proj.weight", "teacher.visual.transformer.resblocks.7.mlp.c_proj.bias", "teacher.visual.transformer.resblocks.7.ln_2.weight", "teacher.visual.transformer.resblocks.7.ln_2.bias", "teacher.visual.transformer.resblocks.8.attn.in_proj_weight", "teacher.visual.transformer.resblocks.8.attn.in_proj_bias", "teacher.visual.transformer.resblocks.8.attn.out_proj.weight", "teacher.visual.transformer.resblocks.8.attn.out_proj.bias", "teacher.visual.transformer.resblocks.8.ln_1.weight", "teacher.visual.transformer.resblocks.8.ln_1.bias", "teacher.visual.transformer.resblocks.8.mlp.c_fc.weight", "teacher.visual.transformer.resblocks.8.mlp.c_fc.bias", "teacher.visual.transformer.resblocks.8.mlp.c_proj.weight", "teacher.visual.transformer.resblocks.8.mlp.c_proj.bias", "teacher.visual.transformer.resblocks.8.ln_2.weight", "teacher.visual.transformer.resblocks.8.ln_2.bias", "teacher.visual.transformer.resblocks.9.attn.in_proj_weight", "teacher.visual.transformer.resblocks.9.attn.in_proj_bias", "teacher.visual.transformer.resblocks.9.attn.out_proj.weight", "teacher.visual.transformer.resblocks.9.attn.out_proj.bias", "teacher.visual.transformer.resblocks.9.ln_1.weight", "teacher.visual.transformer.resblocks.9.ln_1.bias", "teacher.visual.transformer.resblocks.9.mlp.c_fc.weight", "teacher.visual.transformer.resblocks.9.mlp.c_fc.bias", "teacher.visual.transformer.resblocks.9.mlp.c_proj.weight", "teacher.visual.transformer.resblocks.9.mlp.c_proj.bias", "teacher.visual.transformer.resblocks.9.ln_2.weight", "teacher.visual.transformer.resblocks.9.ln_2.bias", "teacher.visual.transformer.resblocks.10.attn.in_proj_weight", "teacher.visual.transformer.resblocks.10.attn.in_proj_bias", "teacher.visual.transformer.resblocks.10.attn.out_proj.weight", "teacher.visual.transformer.resblocks.10.attn.out_proj.bias", "teacher.visual.transformer.resblocks.10.ln_1.weight", "teacher.visual.transformer.resblocks.10.ln_1.bias", "teacher.visual.transformer.resblocks.10.mlp.c_fc.weight", "teacher.visual.transformer.resblocks.10.mlp.c_fc.bias", "teacher.visual.transformer.resblocks.10.mlp.c_proj.weight", "teacher.visual.transformer.resblocks.10.mlp.c_proj.bias", "teacher.visual.transformer.resblocks.10.ln_2.weight", "teacher.visual.transformer.resblocks.10.ln_2.bias", "teacher.visual.transformer.resblocks.11.attn.in_proj_weight", "teacher.visual.transformer.resblocks.11.attn.in_proj_bias", "teacher.visual.transformer.resblocks.11.attn.out_proj.weight", "teacher.visual.transformer.resblocks.11.attn.out_proj.bias", "teacher.visual.transformer.resblocks.11.ln_1.weight", "teacher.visual.transformer.resblocks.11.ln_1.bias", "teacher.visual.transformer.resblocks.11.mlp.c_fc.weight", "teacher.visual.transformer.resblocks.11.mlp.c_fc.bias", "teacher.visual.transformer.resblocks.11.mlp.c_proj.weight", "teacher.visual.transformer.resblocks.11.mlp.c_proj.bias", "teacher.visual.transformer.resblocks.11.ln_2.weight", "teacher.visual.transformer.resblocks.11.ln_2.bias", "teacher.visual.ln_post.weight", "teacher.visual.ln_post.bias", "teacher.transformer.resblocks.0.attn.in_proj_weight", "teacher.transformer.resblocks.0.attn.in_proj_bias", "teacher.transformer.resblocks.0.attn.out_proj.weight", "teacher.transformer.resblocks.0.attn.out_proj.bias", "teacher.transformer.resblocks.0.ln_1.weight", "teacher.transformer.resblocks.0.ln_1.bias", "teacher.transformer.resblocks.0.mlp.c_fc.weight", "teacher.transformer.resblocks.0.mlp.c_fc.bias", "teacher.transformer.resblocks.0.mlp.c_proj.weight", "teacher.transformer.resblocks.0.mlp.c_proj.bias", "teacher.transformer.resblocks.0.ln_2.weight", "teacher.transformer.resblocks.0.ln_2.bias", "teacher.transformer.resblocks.1.attn.in_proj_weight", "teacher.transformer.resblocks.1.attn.in_proj_bias", "teacher.transformer.resblocks.1.attn.out_proj.weight", "teacher.transformer.resblocks.1.attn.out_proj.bias", "teacher.transformer.resblocks.1.ln_1.weight", "teacher.transformer.resblocks.1.ln_1.bias", "teacher.transformer.resblocks.1.mlp.c_fc.weight", "teacher.transformer.resblocks.1.mlp.c_fc.bias", "teacher.transformer.resblocks.1.mlp.c_proj.weight", "teacher.transformer.resblocks.1.mlp.c_proj.bias", "teacher.transformer.resblocks.1.ln_2.weight", "teacher.transformer.resblocks.1.ln_2.bias", "teacher.transformer.resblocks.2.attn.in_proj_weight", "teacher.transformer.resblocks.2.attn.in_proj_bias", "teacher.transformer.resblocks.2.attn.out_proj.weight", "teacher.transformer.resblocks.2.attn.out_proj.bias", "teacher.transformer.resblocks.2.ln_1.weight", "teacher.transformer.resblocks.2.ln_1.bias", "teacher.transformer.resblocks.2.mlp.c_fc.weight", "teacher.transformer.resblocks.2.mlp.c_fc.bias", "teacher.transformer.resblocks.2.mlp.c_proj.weight", "teacher.transformer.resblocks.2.mlp.c_proj.bias", "teacher.transformer.resblocks.2.ln_2.weight", "teacher.transformer.resblocks.2.ln_2.bias", "teacher.transformer.resblocks.3.attn.in_proj_weight", "teacher.transformer.resblocks.3.attn.in_proj_bias", "teacher.transformer.resblocks.3.attn.out_proj.weight", "teacher.transformer.resblocks.3.attn.out_proj.bias", "teacher.transformer.resblocks.3.ln_1.weight", "teacher.transformer.resblocks.3.ln_1.bias", "teacher.transformer.resblocks.3.mlp.c_fc.weight", "teacher.transformer.resblocks.3.mlp.c_fc.bias", "teacher.transformer.resblocks.3.mlp.c_proj.weight", "teacher.transformer.resblocks.3.mlp.c_proj.bias", "teacher.transformer.resblocks.3.ln_2.weight", "teacher.transformer.resblocks.3.ln_2.bias", "teacher.transformer.resblocks.4.attn.in_proj_weight", "teacher.transformer.resblocks.4.attn.in_proj_bias", "teacher.transformer.resblocks.4.attn.out_proj.weight", "teacher.transformer.resblocks.4.attn.out_proj.bias", "teacher.transformer.resblocks.4.ln_1.weight", "teacher.transformer.resblocks.4.ln_1.bias", "teacher.transformer.resblocks.4.mlp.c_fc.weight", "teacher.transformer.resblocks.4.mlp.c_fc.bias", "teacher.transformer.resblocks.4.mlp.c_proj.weight", "teacher.transformer.resblocks.4.mlp.c_proj.bias", "teacher.transformer.resblocks.4.ln_2.weight", "teacher.transformer.resblocks.4.ln_2.bias", "teacher.transformer.resblocks.5.attn.in_proj_weight", "teacher.transformer.resblocks.5.attn.in_proj_bias", "teacher.transformer.resblocks.5.attn.out_proj.weight", "teacher.transformer.resblocks.5.attn.out_proj.bias", "teacher.transformer.resblocks.5.ln_1.weight", "teacher.transformer.resblocks.5.ln_1.bias", "teacher.transformer.resblocks.5.mlp.c_fc.weight", "teacher.transformer.resblocks.5.mlp.c_fc.bias", "teacher.transformer.resblocks.5.mlp.c_proj.weight", "teacher.transformer.resblocks.5.mlp.c_proj.bias", "teacher.transformer.resblocks.5.ln_2.weight", "teacher.transformer.resblocks.5.ln_2.bias", "teacher.transformer.resblocks.6.attn.in_proj_weight", "teacher.transformer.resblocks.6.attn.in_proj_bias", "teacher.transformer.resblocks.6.attn.out_proj.weight", "teacher.transformer.resblocks.6.attn.out_proj.bias", "teacher.transformer.resblocks.6.ln_1.weight", "teacher.transformer.resblocks.6.ln_1.bias", "teacher.transformer.resblocks.6.mlp.c_fc.weight", "teacher.transformer.resblocks.6.mlp.c_fc.bias", "teacher.transformer.resblocks.6.mlp.c_proj.weight", "teacher.transformer.resblocks.6.mlp.c_proj.bias", "teacher.transformer.resblocks.6.ln_2.weight", "teacher.transformer.resblocks.6.ln_2.bias", "teacher.transformer.resblocks.7.attn.in_proj_weight", "teacher.transformer.resblocks.7.attn.in_proj_bias", "teacher.transformer.resblocks.7.attn.out_proj.weight", "teacher.transformer.resblocks.7.attn.out_proj.bias", "teacher.transformer.resblocks.7.ln_1.weight", "teacher.transformer.resblocks.7.ln_1.bias", "teacher.transformer.resblocks.7.mlp.c_fc.weight", "teacher.transformer.resblocks.7.mlp.c_fc.bias", "teacher.transformer.resblocks.7.mlp.c_proj.weight", "teacher.transformer.resblocks.7.mlp.c_proj.bias", "teacher.transformer.resblocks.7.ln_2.weight", "teacher.transformer.resblocks.7.ln_2.bias", "teacher.transformer.resblocks.8.attn.in_proj_weight", "teacher.transformer.resblocks.8.attn.in_proj_bias", "teacher.transformer.resblocks.8.attn.out_proj.weight", "teacher.transformer.resblocks.8.attn.out_proj.bias", "teacher.transformer.resblocks.8.ln_1.weight", "teacher.transformer.resblocks.8.ln_1.bias", "teacher.transformer.resblocks.8.mlp.c_fc.weight", "teacher.transformer.resblocks.8.mlp.c_fc.bias", "teacher.transformer.resblocks.8.mlp.c_proj.weight", "teacher.transformer.resblocks.8.mlp.c_proj.bias", "teacher.transformer.resblocks.8.ln_2.weight", "teacher.transformer.resblocks.8.ln_2.bias", "teacher.transformer.resblocks.9.attn.in_proj_weight", "teacher.transformer.resblocks.9.attn.in_proj_bias", "teacher.transformer.resblocks.9.attn.out_proj.weight", "teacher.transformer.resblocks.9.attn.out_proj.bias", "teacher.transformer.resblocks.9.ln_1.weight", "teacher.transformer.resblocks.9.ln_1.bias", "teacher.transformer.resblocks.9.mlp.c_fc.weight", "teacher.transformer.resblocks.9.mlp.c_fc.bias", "teacher.transformer.resblocks.9.mlp.c_proj.weight", "teacher.transformer.resblocks.9.mlp.c_proj.bias", "teacher.transformer.resblocks.9.ln_2.weight", "teacher.transformer.resblocks.9.ln_2.bias", "teacher.transformer.resblocks.10.attn.in_proj_weight", "teacher.transformer.resblocks.10.attn.in_proj_bias", "teacher.transformer.resblocks.10.attn.out_proj.weight", "teacher.transformer.resblocks.10.attn.out_proj.bias", "teacher.transformer.resblocks.10.ln_1.weight", "teacher.transformer.resblocks.10.ln_1.bias", "teacher.transformer.resblocks.10.mlp.c_fc.weight", "teacher.transformer.resblocks.10.mlp.c_fc.bias", "teacher.transformer.resblocks.10.mlp.c_proj.weight", "teacher.transformer.resblocks.10.mlp.c_proj.bias", "teacher.transformer.resblocks.10.ln_2.weight", "teacher.transformer.resblocks.10.ln_2.bias", "teacher.transformer.resblocks.11.attn.in_proj_weight", "teacher.transformer.resblocks.11.attn.in_proj_bias", "teacher.transformer.resblocks.11.attn.out_proj.weight", "teacher.transformer.resblocks.11.attn.out_proj.bias", "teacher.transformer.resblocks.11.ln_1.weight", "teacher.transformer.resblocks.11.ln_1.bias", "teacher.transformer.resblocks.11.mlp.c_fc.weight", "teacher.transformer.resblocks.11.mlp.c_fc.bias", "teacher.transformer.resblocks.11.mlp.c_proj.weight", "teacher.transformer.resblocks.11.mlp.c_proj.bias", "teacher.transformer.resblocks.11.ln_2.weight", "teacher.transformer.resblocks.11.ln_2.bias". 
	Unexpected key(s) in state_dict: "epoch", "global_step", "pytorch-lightning_version", "state_dict", "callbacks", "optimizer_states", "lr_schedulers", "native_amp_scaling_state". 

How to load the resulting fine-tuned checkpoints properly? I am looking for an advise