Olatunji Ruwase

@luckyq, should this issue remain open? Thanks!

@BlinkDL, thanks for your question. If I understand correctly, it seems there are two parts to this. First, when you say `f(x)` is `float16` and `g(x)` is `float32`, I believe...

@BlinkDL, it is great to hear you have (1) and (2) working. For us to understand the required DeepSpeed support, can you share an example that already incorporates (1) and...

@kiehls90 and @Seong-yeop, nvidia-smi is an imprecise method for tracking memory usage. Can you please use deepspeed's `see_memory_usage()` as described [here](https://github.com/microsoft/DeepSpeed/issues/1437#issuecomment-937981281)? Please share your logs. Thanks!

@kiehls90, in general it is really hard to comment on memory usage without more precise profiling using something like `see_memory_usage()`. My usual approach to debugging is to instrument before/after forward...

@kiehls90, did you make progress with this? Or is it no longer an issue? Thanks!

@kiehls90, is this still an issue, or can we close? Thanks!

@owmohamm, can you please try PR #2145?

@iyupan, thanks for reporting this issue. To help investigate this, can you please provide repro steps? Also, please clarify the expected behavior in this case. Should each parameter gradient be...

@BlinkDL, can you provide details to repro?