Can not download pretrained model for "overall consistency" evaluation

[haoyuhsu]

I've tried the following command to download the pretrained model. wget https://pjlab-gvm-data.oss-cn-shanghai.aliyuncs.com/umt/single_modality/l16_ptk710_ftk710_ftk400_f16_res224.pth -P ~/.cache/vbench/umt_model But I have encountered the following error messages:

> --2024-08-09 21:47:30--  https://pjlab-gvm-data.oss-cn-shanghai.aliyuncs.com/umt/single_modality/l16_ptk710_ftk710_ftk400_f16_res224.pth
> Resolving pjlab-gvm-data.oss-cn-shanghai.aliyuncs.com (pjlab-gvm-data.oss-cn-shanghai.aliyuncs.com)... 47.101.88.10
> Connecting to pjlab-gvm-data.oss-cn-shanghai.aliyuncs.com (pjlab-gvm-data.oss-cn-shanghai.aliyuncs.com)|47.101.88.10|:443... connected.
> HTTP request sent, awaiting response... 403 Forbidden
> 2024-08-09 21:47:31 ERROR 403: Forbidden.

I wonder how to solve this issue or would you kindly provide other sources to download from?

Thank you for your time!

[ziqihuangg]

Thanks for pointing out the issue. In the meantime, you can manually download the model using this temporary link: https://huggingface.co/OpenGVLab/VBench_Used_Models/resolve/main/l16_ptk710_ftk710_ftk400_f16_res224.pth

We're fixing it and will update to the codebase soon. Thanks!

[haoyuhsu]

Thanks!

But I have encountered an error when loading the pre-trained checkpoint. It seems like the state dict does not match.

[haoyuhsu]

After I tried import ViCLIP-L_InternVid-FLT-10M.pth, I still got the following error

args: Namespace(output_path='./evaluation_results/videos-DGE/overall_consistency', full_json_dir='/home/shenlong/Documents/maxhsu/VBench/vbench/VBench_full_info.json', videos_path='/home/shenlong/Documents/maxhsu/all-demo-results/videos-DGE', dimension=['overall_consistency'], load_ckpt_from_local=None, read_frame=None, mode='custom_input', custom_input=False, prompt='', prompt_file=None, category=None, imaging_quality_preprocessing_mode='longer')
start evaluation
Evaluation meta data saved to ./evaluation_results/videos-DGE/overall_consistency/results_2024-08-10-13:57:16_full_info.json
cur_full_info_path: ./evaluation_results/videos-DGE/overall_consistency/results_2024-08-10-13:57:16_full_info.json
2024-08-10 13:57:18,370 - vbench.third_party.ViCLIP.viclip - INFO - Load pretrained weights from /home/shenlong/Downloads/ViCLIP-L_InternVid-FLT-10M.pth
Traceback (most recent call last):
  File "/home/shenlong/Documents/maxhsu/VBench/evaluate.py", line 159, in <module>
    main()
  File "/home/shenlong/Documents/maxhsu/VBench/evaluate.py", line 145, in main
    my_VBench.evaluate(
  File "/home/shenlong/Documents/maxhsu/VBench/vbench/__init__.py", line 150, in evaluate
    results = evaluate_func(cur_full_info_path, self.device, submodules_list, **kwargs)
  File "/home/shenlong/Documents/maxhsu/VBench/vbench/overall_consistency.py", line 59, in compute_overall_consistency
    viclip = ViCLIP(tokenizer= tokenizer, **submodules_list).to(device)
  File "/home/shenlong/Documents/maxhsu/VBench/vbench/third_party/ViCLIP/viclip.py", line 58, in __init__
    state_dict = torch.load(pretrain, map_location='cpu')['model']
  File "/home/shenlong/miniconda3/envs/vbench/lib/python3.10/site-packages/torch/serialization.py", line 789, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/home/shenlong/miniconda3/envs/vbench/lib/python3.10/site-packages/torch/serialization.py", line 1131, in _load
    result = unpickler.load()
  File "/home/shenlong/miniconda3/envs/vbench/lib/python3.10/site-packages/torch/serialization.py", line 1124, in find_class
    return super().find_class(mod_name, name)
ModuleNotFoundError: No module named 'utils.easydict'

[yinanhe]

Sorry for the wrong link. We have just corrected the link. It should be https://huggingface.co/OpenGVLab/VBench_Used_Models/resolve/main/ViClip-InternVid-10M-FLT.pth