Best-Deep-Learning-Optimizers

Best-Deep-Learning-Optimizers copied to clipboard

Hi, I came across this paper today, but they dont have a public implementation. Thought of sharing it here. Thanks for your work!! https://arxiv.org/pdf/2203.01400.pdf 

felipemello1

Thanks for making this nice recap of the hart of AI the optimizers! I have a question regarding the SLS method can you implement something similar to Adam so without...

thegodone

Hi, I'm currently trying DeepMemory, but with latest pytorch (1.7.0), I get the following warning. Do you have any update for DeepMemory? ``` /content/utils/optimizers/deepmemory.py:129: UserWarning: This overload of add_ is...

clementpoiret

Hi, I was wondering what are the benchmarks you used for testing these optimizers? Thanks!

XuezheMax

Any plan to include AdamP?

1

Hi, there is yet another recent proposed optimizer: https://github.com/clovaai/AdamP. Do you plan to include this?

hiyyg

Hi Please use diffGrad_v2.py from https://github.com/shivram1987/diffGrad.

shivram1987

was training a model with cyclic learning rate and after 7th epoch i get nan validation loss..isn't it "exploding gradient problem"? will gradient accumulation be able to solve this issue?...

mobassir94

hv * vi), dim=[2, 3], keepdim=True)) / vi[0, 1].numel() # Hessian diagonal block size is 9 here: torch.sum() reduces the dim 2/3. IndexError: index 1 is out of bounds for...

zzw-zwzhang