Resuming Training from Load Path

[amishparekh]

trafficstars

Hi @mvpatel2000 , This is a question on the shuffling of the dataset. So I was training on a cloud gpu setup and after 40% training I got an OOM error. So to start the training from the current checkpoint, I used the load_path parameter. Now, I am concerned what samples are being put in the next set of training because shuffle was turned on, so is it possible that same samples will be put in and some other samples which have not been used might be completely missed out? Due to lack of credits, the machine has shutdown so I would have to restart on a new machine. I haven't changed the shuffle seed. Thanks