torchtitan icon indicating copy to clipboard operation
torchtitan copied to clipboard

lr scheduler - update global states into optimizer

Open lessw2020 opened this issue 6 months ago • 2 comments

lr scheduler currently maintains two global states to implement the full lr warmup and decay. We want to remove these: "nit: we can make these two arguments still as function arguments below, but we use:

functools.partial to pack these two arguments when adding to the optimizer.

global config works for now. This can be done as a follow up"

lessw2020 avatar Feb 01 '24 18:02 lessw2020

cc: @tianyu-l is this issue done?

awgu avatar Apr 26 '24 15:04 awgu

Seems not. @awgu

https://github.com/pytorch/torchtitan/blob/main/torchtitan/lr_scheduling.py#L10

cc: @tianyu-l is this issue done?

tianyu-l avatar May 03 '24 01:05 tianyu-l