ColossalAI
ColossalAI copied to clipboard
How to set different parameter learning rate for Hybrid Adam?
Thanks for your great work, but could you tell me where I can find the tutorial to adjust different learning rate for different parameter group?
You can just pass the lr attribute when creating a HybridAdam object. Pass different lr values into different parameter groups like adam.