How to reproduce the same training process when using "train_from"

[lemon234071]

Dear,

When the model training was forced to stop due to an accident. I use the opt "train_from" to continue training from the checkpoint. But the result is different from the tranining from start to finish without stopping:

1. The stored patience for "early stop" was not saved into checkpoint.
2. The order of data batch provied train_iter is different, when train_from a checkpoint. （When train_from, it starts over from the begining of the dataset and the data are very different from where it stands at the step of saved checkpoint）

Note that i fixed all random seed.

So it is very convenient that If a reproduction mechanism can be added into the code base. Any help will be greatly appreciated.

[francoishernandez]

This is roughly what I intended with #1826, but it's not compatible with all the changes we introduced in 2.0. It should be possible to introduce such a mechanism though, that would store some counter to keep track of where we are in each dataset. It would never be perfect though, as there is quite a gap between when the data is read and when it's indeed seen in a training batch, because of the pooling mechanism.

[lemon234071]

Thanks for your efforts.

[vince62s]

Regarding this issue I implemented the following: new option -dryrun_steps xxxxx which would batch during xxxxx steps without actually training, then start trianing at xxxxx+1 That would restart the training at the exact point in the data where it stopped. The only issue is that it is very very slow to reach xxxxx+1 If there is a better idea other than storing the index in each dataset.

[PC91]

I made an attempt for this feature in this PR https://github.com/OpenNMT/OpenNMT-py/pull/2520. The idea is to skip to the saved text line in each corpus when a training is resumed.