tiny-cuda-nn icon indicating copy to clipboard operation
tiny-cuda-nn copied to clipboard
verified publisher icon NVlabs

Lion Optimizer

Open JamesPerlman opened this issue 3 years ago • 1 comments

New optimizer, fresh off the press! Chen, et al., 2023. "Symbolic Discovery of Optimization Algorithms" describes the Evolved Sign Momentum optimizer, or "Lion."

I had to try it out with NeRF but I haven't gotten great results yet. The config file I added here did give me some good results with mlp_learning_an_image that appear to rival Adam, but I haven't done extensive benchmarking.

Code was rewritten from https://github.com/google/automl/blob/master/lion/lion_pytorch.py - enjoy!

JamesPerlman avatar Feb 16 '23 09:02 JamesPerlman

Hi, thank you for the contribution -- that was very fast! :)

I'm afraid I don't have time to review right now, but should be able to get to it ~next week.

Tom94 avatar Feb 16 '23 11:02 Tom94