pytorch-ssd icon indicating copy to clipboard operation
pytorch-ssd copied to clipboard

Added Nesterov momentum to SGD optimizer

Open georgekasa opened this issue 1 year ago • 1 comments

georgekasa avatar Apr 09 '23 14:04 georgekasa

I added a """feature"""" if someone could add Nesterov momentum in SGD opt, train_ssd.py -> line 375 optimizer = torch.optim.SGD(params, lr=args.lr, momentum=args.momentum, weight_decay=args.weight_decay, nesterov = True)

as Karpathy told in CS231n: Nesterov Momentum is a slightly different version of the momentum update that has recently been gaining popularity. It enjoys stronger theoretical converge guarantees for convex functions and in practice it also consistenly works slightly better than standard momentum.

georgekasa avatar Apr 09 '23 14:04 georgekasa