Knet.jl icon indicating copy to clipboard operation
Knet.jl copied to clipboard

Published 20 hours ago • verified publisher icon denizyuret

training API improvements

Open denizyuret opened this issue 4 years ago • 2 comments

  • [ ] SGD learning rate scheduler
  • [ ] Global gradient clip
  • [ ] Default to override param.opt rather than keep

denizyuret avatar Oct 23 '19 06:10 denizyuret

See also #564

denizyuret avatar Jul 25 '20 09:07 denizyuret

In the xnornet paper (https://arxiv.org/pdf/1603.05279.pdf pp. 7), LR update follows parameter update every iteration. We can generalize this to any optimizer update (maybe we want to change more than just LR) and run it every iteration. What inputs would such an update need? Iteration number? Loss values?

denizyuret avatar Mar 19 '21 06:03 denizyuret