Boxiang Wang

icon indicating a website link https://boxiangw.github.io/

icon company @NVIDIA icon location Sunnyvale, CA icon location There are no solutions, there are only trade-offs.

Results 30 comments of Boxiang Wang

MuonClip support (non-split version)

/ok to test 55cc00d

MuonClip support (non-split version)

/ok to test 9095615

MuonClip support (non-split version)

/ok to test 1bb0407

MuonClip support (non-split version)

/ok to test b63c573

MuonClip support (non-split version)

/ok to test 95fdba3

MuonClip support (non-split version)

/ok to test 6562a52

[Dev] docs(megatron-fsdp): add Megatron-FSDP user guide

Thanks Xuwen for this great documentation, can you also move it or at least link it [here ](https://docs.nvidia.com/megatron-core/developer-guide/latest/api-guide/custom_fsdp.html) ([code](https://github.com/NVIDIA/Megatron-LM/blob/main/docs/source/api-guide/custom_fsdp.md))? Not sure where is this doc/discussion being published.

Add muon and layerwise distributed optimizer

@JavaZeroo simply set `--optimizer dist_muon`(Layerwise distributed optimizer) or `--optimizer muon` will turn on Muon usage

Long context performance doc hot fix

LGTM, thanks

RAM Usage Kept Increasing

Hi @dimapihtar, can you help take a look? Thanks