NotImplementedError: Cannot copy out of meta tensor; no data!

[TerryYiDa]

当我训练的时候出现这个问题？请问怎么解决。

[Yingshu-Li]

same error.

[Roberyan]

I have this error after I upgrade transformers from 4.31 to 4.35,

Traceback (most recent call last):
  File "/home/user/LISA/train_ds.py", line 589, in <module>
    main(sys.argv[1:])
  File "/home/user/LISA/train_ds.py", line 310, in main
    model_engine, optimizer, train_loader, scheduler = deepspeed.initialize(
  File "/home/user/anaconda3/envs/LISA/lib/python3.9/site-packages/deepspeed/__init__.py", line 181, in initialize
    engine = DeepSpeedEngine(args=args,
  File "/home/user/anaconda3/envs/LISA/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 262, in __init__
    self._configure_distributed_model(model)
  File "/home/user/anaconda3/envs/LISA/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1112, in _configure_distributed_model
    self.module.to(self.device)
  File "/home/user/anaconda3/envs/LISA/lib/python3.9/site-packages/torch/nn/modules/module.py", line 989, in to
    return self._apply(convert)
  File "/home/user/anaconda3/envs/LISA/lib/python3.9/site-packages/torch/nn/modules/module.py", line 641, in _apply
    module._apply(fn)
  File "/home/user/anaconda3/envs/LISA/lib/python3.9/site-packages/torch/nn/modules/module.py", line 641, in _apply
    module._apply(fn)
  File "/home/user/anaconda3/envs/LISA/lib/python3.9/site-packages/torch/nn/modules/module.py", line 641, in _apply
    module._apply(fn)
  [Previous line repeated 1 more time]
  File "/home/user/anaconda3/envs/LISA/lib/python3.9/site-packages/torch/nn/modules/module.py", line 664, in _apply
    param_applied = fn(param)
  File "/home/user/anaconda3/envs/LISA/lib/python3.9/site-packages/torch/nn/modules/module.py", line 987, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
NotImplementedError: Cannot copy out of meta tensor; no data!

Does anyone know how to solve this without degrading the transformers version