ColossalAI icon indicating copy to clipboard operation
ColossalAI copied to clipboard

How to set different parameter learning rate for Hybrid Adam?

Open crepuscularlight opened this issue 6 months ago • 2 comments

crepuscularlight avatar Jul 10 '25 08:07 crepuscularlight

Thanks for your great work, but could you tell me where I can find the tutorial to adjust different learning rate for different parameter group?

crepuscularlight avatar Jul 10 '25 08:07 crepuscularlight

You can just pass the lr attribute when creating a HybridAdam object. Pass different lr values into different parameter groups like adam.

flybird11111 avatar Aug 19 '25 09:08 flybird11111