Feature request: Please, add implementation for Novograd algorithm

[LSinev]

Great work!

Can you, please, add implementation for Novograd algorithm? Support info: paper: https://arxiv.org/abs/1905.11286

Novograd implementations: https://github.com/NVIDIA/apex/blob/master/apex/optimizers/fused_novograd.py https://github.com/jettify/pytorch-optimizer/blob/master/torch_optimizer/novograd.py https://github.com/convergence-lab/novograd https://github.com/lonePatient/NovoGrad-pytorch https://github.com/titu1994/keras_novograd

[TimDettmers]

I had a look before at this. It is a bit more work and I will probably not focus on it for the next release, but it's good to know that people are interested in this (I had the request before). I will probably implement it sometime next month.