DeepSpeed icon indicating copy to clipboard operation
DeepSpeed copied to clipboard

enable apex.optimizers.FusedAdam and deepspeed fused fp16 optimizer

Open ShadenSmith opened this issue 4 years ago • 1 comments

If a client provides an optimizer of type apex.optimizers.FusedAdam, it won't be given a fused FP16 optimizer. The isinstance() query here only checks for DeepSpeed's FusedAdam.

The implementation here was just hacked together to test the logic. We could also test for apex.optimizers.FusedAdam specifically, and just be careful not handle missing apex?

Would also love to put together a list of requirements for compatibility!

Major question: can we merge this without breaking existing checkpoints? We may need to handle loading from an unfused fp16 optimizer checkpoint.

ShadenSmith avatar Feb 04 '21 08:02 ShadenSmith

Can one of the admins verify this patch?

rocm-mici avatar Jun 09 '22 20:06 rocm-mici