ColossalAI icon indicating copy to clipboard operation
ColossalAI copied to clipboard
verified publisher icon hpcaitech

[FEATURE]: add master_weights arg to HybridParallelPlugin

Open eiPI1-0 opened this issue 5 months ago • 1 comments

Describe the feature

When using CPU offload, setting master_weights=False in both GeminiPlugin and LowLevelZeroPlugin can reduce GPU memory usage and improve speed. Does HybridParallelPlugin also support this feature?

eiPI1-0 avatar Jul 23 '25 07:07 eiPI1-0

Zero Optimizer usually updates parameters using float32. Not using float32 may lead to unstable training.

flybird11111 avatar Aug 12 '25 05:08 flybird11111