tinygrad icon indicating copy to clipboard operation
tinygrad copied to clipboard
verified publisher icon tinygrad

LR Schedulers

Open mattismegevand opened this issue 2 years ago • 4 comments

Added three learning rate scheduler:

  • MultiStepLR
  • ReduceLROnPlateau
  • CosineAnnealingLR

mattismegevand avatar Apr 21 '23 20:04 mattismegevand

Put test in test/extra, and can you add a couple integration tests that train models with these schedulers?

geohot avatar May 04 '23 22:05 geohot

Done

mattismegevand avatar May 07 '23 16:05 mattismegevand

Tests failing

geohot avatar May 25 '23 23:05 geohot

Forgot to set the seed for tensors, should be good now

mattismegevand avatar May 26 '23 17:05 mattismegevand

Why did it fail with SGD? Is there a reason or just flakiness?

geohot avatar May 27 '23 00:05 geohot

There’s no reason in particular both optimizers works with the schedulers. Since for the testing we're training on MNIST it was easy for the model to reach convergence without a scheduler. So I tried multiple settings and Adam with those parameters worked more reliably and could pass the 3 categories of test (CPU / LLVM / Torch).

Less shaky testing could be done by training on ImageNet for example but I preferred to do something lightweight that didn't require lots of training time.

mattismegevand avatar May 27 '23 08:05 mattismegevand

Cool, merged!

geohot avatar May 27 '23 14:05 geohot