sd-scripts icon indicating copy to clipboard operation
sd-scripts copied to clipboard

Implement Unit-Scaled Maximal Update Parametrization for best LR

Open StableLlama opened this issue 6 months ago • 0 comments

There is a new paper, "u-μP: The Unit-Scaled Maximal Update Parametrization" at https://arxiv.org/abs/2407.17465 that promises selection of optimal hyperparameters, like the learning rate, even when using very small datatypes like FP8

The content is too advanced for me, but I had the impression that it might be a valuable addition to kohya_ss

StableLlama avatar Jul 25 '24 18:07 StableLlama