Question about Lion

[kellenf]

Thank your for your paper and work, but I meet some wired problem in training. Can you help me? It's my training process curve. Notice Lion will increase loss sometimes but decease later. It seems not stability? Can you give me some advises how to tune Lion's parameters? Below is my setting: Batchsize = 8, GPU = 64 A100, with syncBN Network: PointPillars Optimizer setting:

[xiangning-chen]

Thanks for the data point. Can you please tell us your baseline settings? Like which optimizer, which learning rate and weight decay?

I think for instability problem, you can either reduce the learning rate a little bit or try to lower the beta2, for example, we used (0.95, 0.98) in our llm training.