Megatron-LM icon indicating copy to clipboard operation
Megatron-LM copied to clipboard
Published 1 month ago • verified publisher icon NVIDIA

Build dataset for all GPUs with tp_rank=0 and pp_rank=0 or -1 in multi-machine training.

Open wan-nan opened this issue 9 months ago • 1 comments

Not just for the GPU with global_rank=0, following the code below. https://github.com/NVIDIA/Megatron-LM/blob/28118fcdc22e42621776a021af568ae39c198418/pretrain_gpt.py#L257-L260 https://github.com/NVIDIA/Megatron-LM/blob/28118fcdc22e42621776a021af568ae39c198418/pretrain_gpt.py#L306-L311

wan-nan avatar Mar 14 '25 03:03 wan-nan

Marking as stale. No activity in 60 days.

github-actions[bot] avatar May 13 '25 18:05 github-actions[bot]

This PR was closed because it has been inactive for 7 days since being marked as stale.

github-actions[bot] avatar Jul 28 '25 02:07 github-actions[bot]