Zijie Yan comments

Results 58 comments of


                                            Zijie Yan

Hybrid Data x Context Parallelism Feature

Hi @parthmannan, could you also start a main PR?

[BUG]Error in get_transformer_layer_offset when virtual_pipeline=True, num_layers_in_last_pipeline_stage is set, and num_layers_in_first_pipeline_stage is not

Hey @yqy3214, sorry for the late update. We've recently added a new feature to handle uneven splits more effectively, with a customizable layout. Could you please check this link? It...

[Dev][Draft] fix(moe): Support HybridEP and reduce memory overhead for 1F1B A2A overlap

Thanks for the PR. Please mark the title with [Dev] fix(moe): xxx and label this PR with `module:moe` and `dev`.

[Dev][Draft] fix(moe): Support HybridEP and reduce memory overhead for 1F1B A2A overlap

/ok to test 6102cc5

[Dev][Draft] fix(moe): Support HybridEP and reduce memory overhead for 1F1B A2A overlap

/ok to test d29b634

[Dev][Draft] fix(moe): Support HybridEP and reduce memory overhead for 1F1B A2A overlap

@lhb8125 could you fix the api checks? https://github.com/NVIDIA/Megatron-LM/blob/main/docs/api-backwards-compatibility-check.md

[Dev] feat: hybrid-cp for dev branch (part 2)

Is there any difference between this and https://github.com/NVIDIA/Megatron-LM/pull/2054?

[Dev] feat: hybrid-cp for dev branch (part 2)

> > Is there any difference between this and #2054? > > This is the second MR, we need to merge 2054 first, and then this 2000 (The reason the...

[Dev] Changes to support multimodule pipelining

Hi @yashaswikarnati, please include the main PR link in the desc.

[Dev] Changes to support multimodule pipelining

> @yanring @lhb8125 gentle reminder on the review Thanks for the reminder. I'm actually waiting for a review from @lhb8125 and @shifangx to help take a look