nanoGPT
nanoGPT copied to clipboard
Resume Training
https://github.com/karpathy/nanoGPT/blob/325be85d9be8c81b436728a420e85796c57dba7e/train.py#L106
In my implementation of the code, I modified this line to incorporate the iteration into the seed. I suspect that if you resume training multiple times, the random seed may draw the same training sets and in the same order. Over many iterations of the dataset this may be lost, but it may cause a pattern of validating against the same sets as well, which has higher consequences.