Zijie Yan comments

Results 58 comments of


                                            Zijie Yan

[Dev] docs(megatron-fsdp): add Megatron-FSDP user guide

#### `docs/discussions/megatron-fsdp-user-guide/example-scripts/sbatch_mfsdp_deepseek_v3.sh` **Line 1:** > 🔴 **Critical**: The shebang uses a full-width exclamation mark (`#！/bin/bash`) instead of ASCII (`#!/bin/bash`). This will prevent the script from executing properly. Please replace the...

[Dev] docs(megatron-fsdp): add Megatron-FSDP user guide

/ok to test 5139bcda05f3f1ad554f10a7763062c0a00808c7

Changes to support latent MoEs

Thanks for the work. Could you also add UT and integration tests covering this feature combined with EP/TP?

[Dev] Feature: linear cross entropy fusion

@lhb8125 Hey Hongbin, could you help run the functional test on GitLab for this PR?

[BUG] Multi-Head-Latent-Attention Error

Thanks for reporting the issue, could you share your TE version?

[training migration] add training config dataclass and arg generation utility

Hey @maanug-nv @ericharper, do we have any design doc for this?

Remove redundant reduce in aux_loss logging

Please correctly label this PR

Remove redundant reduce in aux_loss logging

/ok to test fe8e9e26f1e939f96a8240cda7d6841d582458df

[BUG] MoE load balancing loss is accumulated twice when using activation checkpointing

Thanks for reporting and fixing this, this is likely a display bug with no impact on convergence. We'll take your PR internally and help get it merged.

[BUG] MoE load balancing loss is accumulated twice when using activation checkpointing

Hi @thuwzt, we've merged the fix with you as co-author in May. Sorry for the late notice, and thanks for your contribution! https://github.com/NVIDIA/Megatron-LM/commit/e6d56d6828c0773f55772b92b2ec0eed5639665e