lit-llama Fixing the pretrain script for Loss Averaging and no_backward

Fixing the pretrain script for Loss Averaging and no_backward_sync()

Open LamOne1 opened this issue 1 year ago • 2 comments

I think #357 should be applied to the pretrain script as well.

Thank you so much lightning team for this amazing repository.

Jun 08 '23 13:06 LamOne1

The red pajama pretrain script has gradient accumulation already. Shakespeare is missing it, it could be added too there yes. Contributions welcome!

Jun 08 '23 14:06 awaelchli

I'm not yet familiar with GitHub and the code editing, but I'm eager to learn and help out! I'll try to learn and edit Shakespeare code

Jun 10 '23 12:06 LamOne1