DeepSpeed
                                
                                
                                
                                    DeepSpeed copied to clipboard
                            
                            
                            
                        enable apex.optimizers.FusedAdam and deepspeed fused fp16 optimizer
If a client provides an optimizer of type apex.optimizers.FusedAdam, it won't be given a fused FP16 optimizer. The isinstance() query here only checks for DeepSpeed's FusedAdam.
The implementation here was just hacked together to test the logic. We could also test for apex.optimizers.FusedAdam specifically, and just be careful not handle missing apex?
Would also love to put together a list of requirements for compatibility!
Major question: can we merge this without breaking existing checkpoints? We may need to handle loading from an unfused fp16 optimizer checkpoint.
Can one of the admins verify this patch?