Megatron-LM
Megatron-LM copied to clipboard
Reload optimizer params when loading bf16 ckpt
This PR is to fix this issue, when using bf16 and attempting to load pretrained model weights to continue training, the weights in the optimizer will not be reset to the pretrained model weights.