GaLore icon indicating copy to clipboard operation
GaLore copied to clipboard

Published 20 hours ago • verified publisher icon jiaweizzhao

Why not reproject the internal Adam states during update_proj_gap?

Open liuliu opened this issue 7 months ago • 2 comments

Hi, great project. After reading the paper and the implementation, I am wondering if it is considered to reproject the Adam internal states (exp_avg, exp_avg_sq) from previous subspace to the new subspace?

liuliu avatar Jun 30 '24 16:06 liuliu