How can I resume training from the dataset point where the model left off

[Nuri-Tas]

❓ Questions and Help

I have 35GB data and I'm unable to pretrain RoBERTa in one go. Is it possible to stop training and then continue it from the same data point the model left off? I know the fairseq searches for the last checkpoint to resume training, but I'm mainly interested in proceeding training from the exact same datapoint.

What's your environment?

fairseq Version: 0.12.2 PyTorch Version: 1.12.1+cu113 OS: Linux How you installed fairseq: source (git clone) Python version: 3.8.8 CUDA version: 11.4 GPU models and configuration: 24GB RTX3090