sd-scripts
sd-scripts copied to clipboard
Implement Unit-Scaled Maximal Update Parametrization for best LR
There is a new paper, "u-μP: The Unit-Scaled Maximal Update Parametrization" at https://arxiv.org/abs/2407.17465 that promises selection of optimal hyperparameters, like the learning rate, even when using very small datatypes like FP8
The content is too advanced for me, but I had the impression that it might be a valuable addition to kohya_ss