Hongbin Liu comments

Results 28 comments of


                                            Hongbin Liu

建议根据小爱同学推荐的播放列表下载并播放歌曲

+1 非常好的 feature

[BUG] Loss difference when training with FP8 vs. BF16 MoE

Try disabling the recomputation.

[BUG] Loss difference when training with FP8 vs. BF16 MoE

I am not sure if disabling recomputation does work in your case, maybe you can double-check it by enabling recomputation and using 16 layers. I think the loss diff is...

Split wgrad&dgrad from backward() to support a2a overlap

> @lhb8125 could you please sign off your commits ([guide](https://github.com/NVIDIA/TransformerEngine/blob/main/CONTRIBUTING.rst#sign-your-work))? @ksivaman Changing the history is too tricky. Could you turn to [this MR](https://github.com/NVIDIA/TransformerEngine/pull/1653), where all commits are signed off?

[错误报告]: Slack 无法发送下载开始消息

Any comment?

[错误报告]: Slack 无法发送下载开始消息

@jxxghp

Feature: linear cross entropy fusion

> Did you try out storing the logits in bf16? It could save lot of memory. Not sure if we need this fusion. @kvareddy Jack has given a good case...

[Draft] fix(moe): Support HybridEP and reduce memory overhead for 1F1B A2A overlap

/ok to test 7073437

[Draft] fix(moe): Support HybridEP and reduce memory overhead for 1F1B A2A overlap

/ok to test 619bb51

[Draft] fix(moe): Support HybridEP and reduce memory overhead for 1F1B A2A overlap

/ok to test aac464e