Can't load CLIP-ViT-H-14-frozen-xlm-roberta-large-laion5B-s13B-b90k

[kimihailv]

When I am trying to load this model, the following error occurred:

import open_clip
model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms('hf-hub:laion/CLIP-ViT-H-14-frozen-xlm-roberta-large-laion5B-s13B-b90k')
Downloading (…)ip_pytorch_model.bin: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.77G/4.77G [02:20<00:00, 33.9MB/s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/huggingface_hub/utils/_errors.py", line 261, in hf_raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/laion/CLIP-ViT-H-14-frozen-xlm-roberta-large-laion5B-s13B-b90k/resolve/main/open_clip_config.json

[kimihailv]

The same for laion/CLIP-ViT-B-32-xlm-roberta-base-laion5B-s13B-b90k

[kimihailv]

if I try to load model without HF, this error encountered:

import open_clip
model, _, preprocess = open_clip.create_model_and_transforms('xlm-roberta-large-ViT-H-14', pretrained='frozen_laion5b_s13b_b90k')
Downloading (…)lve/main/config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 616/616 [00:00<00:00, 136kB/s]
Downloading model.safetensors: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.24G/2.24G [01:05<00:00, 34.4MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/open_clip/factory.py", line 308, in create_model_and_transforms
    model = create_model(
  File "/usr/local/lib/python3.8/dist-packages/open_clip/factory.py", line 228, in create_model
    load_checkpoint(model, checkpoint_path)
  File "/usr/local/lib/python3.8/dist-packages/open_clip/factory.py", line 104, in load_checkpoint
    incompatible_keys = model.load_state_dict(state_dict, strict=strict)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for CustomTextCLIP:
        Unexpected key(s) in state_dict: "text.transformer.embeddings.position_ids". 

[dcferreira]

Downgrading my transformers version worked for me.

pip install -U transformers==4.30.2

[EIFY]

I also just put out a PR yesterday to fix this: https://github.com/mlfoundations/open_clip/pull/595

if I try to load model without HF, this error encountered:

import open_clip
model, _, preprocess = open_clip.create_model_and_transforms('xlm-roberta-large-ViT-H-14', pretrained='frozen_laion5b_s13b_b90k')
Downloading (…)lve/main/config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 616/616 [00:00<00:00, 136kB/s]
Downloading model.safetensors: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.24G/2.24G [01:05<00:00, 34.4MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/open_clip/factory.py", line 308, in create_model_and_transforms
    model = create_model(
  File "/usr/local/lib/python3.8/dist-packages/open_clip/factory.py", line 228, in create_model
    load_checkpoint(model, checkpoint_path)
  File "/usr/local/lib/python3.8/dist-packages/open_clip/factory.py", line 104, in load_checkpoint
    incompatible_keys = model.load_state_dict(state_dict, strict=strict)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for CustomTextCLIP:
        Unexpected key(s) in state_dict: "text.transformer.embeddings.position_ids". 

[kimihailv]

What about first error, when config can't be found?

[rwightman]

no open_clip_config.json was pushed by whoever uploaded this model, so the hf-hub method won't work as it sourced the model config from the hub instead of open_clip...

[zhangluustb]

use this instead: model, _, preprocess = open_clip.create_model_and_transforms('xlm-roberta-large-ViT-H-14', pretrained='frozen_laion5b_s13b_b90k') tokenizer = open_clip.get_tokenizer('xlm-roberta-large-ViT-H-14')