nanoGPT icon indicating copy to clipboard operation
nanoGPT copied to clipboard

Resume Training

Open tiredsoul21 opened this issue 3 months ago • 3 comments

https://github.com/karpathy/nanoGPT/blob/325be85d9be8c81b436728a420e85796c57dba7e/train.py#L106

In my implementation of the code, I modified this line to incorporate the iteration into the seed. I suspect that if you resume training multiple times, the random seed may draw the same training sets and in the same order. Over many iterations of the dataset this may be lost, but it may cause a pattern of validating against the same sets as well, which has higher consequences.

tiredsoul21 avatar Mar 27 '24 12:03 tiredsoul21