Feature request: is Adam-mini optimizer worth adding?

[jim-plus]

Reminder

• [X] I have read the README and searched the existing issues.

System Info

N/A

Reproduction

N/A

Expected behavior

Consideration of Adam-mini optimizer, which claims to be able to significantly boost throughput for both training and pretraining while reducing memory consumption. https://arxiv.org/abs/2406.16793

Others

No response