Open-Sora-Plan icon indicating copy to clipboard operation
Open-Sora-Plan copied to clipboard

Published 20 hours ago • verified publisher icon PKU-YuanGroup

GaLore optimizer

Open ostix360 opened this issue 11 months ago • 1 comments

Hi!

Galore optimiser is an optimiser based on Adam that projects the gradient, so the optimiser memory is reduced and the gradient memory is null (or near 0).

See the paper for more information

This optimiser seems promising and can be usefull to train this kind of big model

ostix360 avatar Mar 26 '24 08:03 ostix360

Thank you for your advice! We've included it as a future plan.

LinB203 avatar Mar 28 '24 10:03 LinB203