DemonRangerOptimizer icon indicating copy to clipboard operation
DemonRangerOptimizer copied to clipboard

Published 20 hours ago • verified publisher icon JRC1995
→

Metadata

Quasi Hyperbolic Rectified DEMON Adam/Amsgrad with AdaMod, Gradient Centralization, Lookahead, iterative averaging and decorrelated Weight Decay

Results 2 DemonRangerOptimizer issues
Sort by recently updated
recently updated
newest added

Recent pytorch's adam changes

https://github.com/pytorch/pytorch/commits/master/torch/optim/adam.py Where is a lot of changes like no grads cope decorator. Not sure if it affects math/performance. For your consideration.

hadaev8 avatar hadaev8

Instability with HyperProp

39 comment

I tried it in two tasks, but got nans during training, any suggestions?

hadaev8 avatar hadaev8

← Metadata

25
Stars
6
Forks
Watchers

Owner

verified publisher icon JRC1995

Metadata

Quasi Hyperbolic Rectified DEMON Adam/Amsgrad with AdaMod, Gradient Centralization, Lookahead, iterative averaging and decorrelated Weight Decay

Back