Diffusion-Video-Autoencoders icon indicating copy to clipboard operation
Diffusion-Video-Autoencoders copied to clipboard

FineTune Model

Open mdv3101 opened this issue 1 year ago • 0 comments

I am trying to fine-tune the model on a different face dataset. However I am getting the following error.

Traceback (most recent call last):
  File "run_train.py", line 8, in <module>
    train(conf, gpus=gpus, nodes=nodes)
  File "/home/Diffusion-Video-Autoencoders/experiment.py", line 769, in train
    trainer.fit(model)
  File "/home/env_tf/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 552, in fit
    self._run(model)
  File "/home/env_tf/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 914, in _run
    self.checkpoint_connector.restore_training_state()
  File "/home/env_tf/lib/python3.7/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py", line 164, in restore_training_state
    self.restore_loops()
  File "/home/env_tf/lib/python3.7/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py", line 217, in restore_loops
    self.trainer.fit_loop.load_state_dict(state_dict["fit_loop"])
  File "/home/env_tf/lib/python3.7/site-packages/pytorch_lightning/loops/base.py", line 191, in load_state_dict
    v.load_state_dict(state_dict.copy(), prefix + k + ".", restart_progress)
  File "/home/env_tf/lib/python3.7/site-packages/pytorch_lightning/loops/base.py", line 191, in load_state_dict
    v.load_state_dict(state_dict.copy(), prefix + k + ".", restart_progress)
  File "/home/env_tf/lib/python3.7/site-packages/pytorch_lightning/loops/base.py", line 188, in load_state_dict
    self._load_from_state_dict(state_dict.copy(), prefix, restart_progress)
  File "/home/env_tf/lib/python3.7/site-packages/pytorch_lightning/loops/base.py", line 196, in _load_from_state_dict
    v.load_state_dict(state_dict[prefix + k])
KeyError: 'epoch_loop.batch_loop.optim_progress'

Is there an issue with the checkpoint file?

mdv3101 avatar Apr 13 '23 01:04 mdv3101