[FEATURE]: Expotential Moving Average (EMA)

Open taoisu opened this issue 3 years ago • 1 comments

Describe the feature

The EMA is used to train many models, an example can be found here: https://github.com/CompVis/latent-diffusion/blob/main/ldm/modules/ema.py

But with model sharding it is unclear how to implement ema properly. It would be really nice to have a sample to showcase how to implement ema properly in Colossal AI.

Jul 13 '22 21:07 taoisu

Hi @taoisu , thanks for your idea. We have recorded your suggestion and will prioritize it later.

Jul 14 '22 08:07 binmakeswell

https://github.com/hpcaitech/ColossalAI/blob/main/examples/images/diffusion/ldm/models/autoencoder.py#L11 We have updated a lot. This issue was closed due to inactivity. Thanks.

Apr 13 '23 04:04 binmakeswell