sd-scripts icon indicating copy to clipboard operation
sd-scripts copied to clipboard

Published 20 hours ago • verified publisher icon kohya-ss

Support GaLore Optimazer

Open alfredplpl opened this issue 11 months ago • 2 comments

The optimizer is memory efficient. We can pretrain mistral-7B with 24GB.

https://github.com/jiaweizzhao/GaLore

image

alfredplpl avatar Mar 07 '24 06:03 alfredplpl

Sorry. It does not have feasibly. It takes many days.

alfredplpl avatar Mar 07 '24 16:03 alfredplpl

Well, this thing is a bit tricky to implement, and you need to get the weights for each layer individually, instead of just adding an optimizer

sdbds avatar Mar 20 '24 09:03 sdbds