Model load error in Ola model

[zhizhou57]

When I try to evaluate Ola model on MMMU with such command: python run.py --model ola --data MMMU_DEV_VAL I encounter this error when loading speech model:

[2025-04-09 06:46:03] ERROR - run.py: main - 433: Model ola x Dataset MMMU_DEV_VAL combination failed: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device., skipping this combination.
Traceback (most recent call last):
  File "/user/liuyesheng/VLMEvalKit/run.py", line 317, in main
    model = infer_data_job(
  File "/user/liuyesheng/VLMEvalKit/vlmeval/inference.py", line 176, in infer_data_job
    model = infer_data(
  File "/user/liuyesheng/VLMEvalKit/vlmeval/inference.py", line 110, in infer_data
    model = supported_VLM[model_name]() if isinstance(model, str) else model
  File "/user/liuyesheng/VLMEvalKit/vlmeval/vlm/ola/ola_model.py", line 40, in __init__
    self.tokenizer, self.model, self.image_processor, self.context_len = load_pretrained_model(
  File "/user/liuyesheng/VLMEvalKit/vlmeval/vlm/ola/ola/model/builder.py", line 63, in load_pretrained_model
    model = model_cls.from_pretrained(
  File "/root/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 279, in _wrapper
    return func(*args, **kwargs)
  File "/root/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4342, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/user/liuyesheng/VLMEvalKit/vlmeval/vlm/ola/ola/model/language_model/ola_qwen.py", line 35, in __init__
    self.model = OlaQwenModel(config)
  File "/user/liuyesheng/VLMEvalKit/vlmeval/vlm/ola/ola/model/language_model/ola_qwen.py", line 25, in __init__
    super(OlaQwenModel, self).__init__(config)
  File "/user/liuyesheng/VLMEvalKit/vlmeval/vlm/ola/ola/model/ola_arch.py", line 22, in __init__
    self.speech_encoder = build_speech_encoder(config)
  File "/user/liuyesheng/VLMEvalKit/vlmeval/vlm/ola/ola/model/speech_encoder/builder.py", line 9, in build_speech_encoder
    return DualWrappedEncoder(config)
  File "/user/liuyesheng/VLMEvalKit/vlmeval/vlm/ola/ola/model/speech_encoder/speech_encoder.py", line 34, in __init__
    self.whisper_model = self.load_whisper(config)
  File "/user/liuyesheng/VLMEvalKit/vlmeval/vlm/ola/ola/model/speech_encoder/speech_encoder.py", line 50, in load_whisper
    encoder = whisper.load_model(name=model_config.speech_encoder, device='cuda').encoder
  File "/home/liuyesheng/miniconda3/envs/ola/lib/python3.10/site-packages/whisper/__init__.py", line 162, in load_model
    return model.to(device)
  File "/home/liuyesheng/miniconda3/envs/ola/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1343, in to
    return self._apply(convert)
  File "/home/liuyesheng/miniconda3/envs/ola/lib/python3.10/site-packages/torch/nn/modules/module.py", line 903, in _apply
    module._apply(fn)
  File "/home/liuyesheng/miniconda3/envs/ola/lib/python3.10/site-packages/torch/nn/modules/module.py", line 903, in _apply
    module._apply(fn)
  File "/home/liuyesheng/miniconda3/envs/ola/lib/python3.10/site-packages/torch/nn/modules/module.py", line 930, in _apply
    param_applied = fn(param)
  File "/home/liuyesheng/miniconda3/envs/ola/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1336, in convert
    raise NotImplementedError(
NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.