yoyodyne
yoyodyne copied to clipboard
Epoch-based warmup
Standard procedure for LR schedulers is to step once per epoch unless otherwise specified. We have one exception to this: the warmup inverse-square-root scheduler.
This is sort of important in the context of #60 aka the LightingCLI migration. If we use the inherited optimization routine (and we otherwise can) there's no way to pass the information that the warmup inverse-square-root scheduler is a step-based one, unless we override the optimization stepping and hardcode a check (is this a WarmupInverseSquareRootSchedule
object?) into the base model. Rather than doing this, I propose we just change the scheduler to work with epochs rather than steps.