Shijie Wu

Thanks! I haven't tried to inject it yet. But I assume it would look something like [this](https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/module_inject/replace_policy.py) unless it's not supported, in which case it would look like the example...

sorry for the delay! PTAL @muellerzr @mjbommar

Gentle ping @muellerzr @mjbommar :)

@152334H it does convert grad_norm to number before passing it into `_maybe_log_save_evaluate` https://github.com/huggingface/transformers/blob/831bc25d8fdb85768402f772cf65cc3d7872b211/src/transformers/trainer.py#L2010-L2016 same for deepspeed https://github.com/microsoft/DeepSpeed/blob/bcc617a0009dd27b4e144de59979bd7770eaf57c/deepspeed/runtime/engine.py#L448-L458 what backend were you using?