cruise icon indicating copy to clipboard operation
cruise copied to clipboard

Introduce momentum to DNN

Open gyeongin opened this issue 8 years ago • 1 comments

A lot of researches articulate that momentum may accelerate job convergence of DNN. We should introduce momentum to our DNN codebase.

gyeongin avatar Sep 27 '16 08:09 gyeongin

  • (Apparently) the first paper to propose the momentum method: Polyak, "Some methods of speeding up the convergence of iteration methods", 1964.
  • A good comparison of the classical momentum method and the Nesterov version: http://www.cs.toronto.edu/~hinton/absps/momentum.pdf, Section 2
  • Very short version: http://cs231n.github.io/neural-networks-3/#sgd

jsjason avatar Nov 24 '16 00:11 jsjason