Megatron-LM icon indicating copy to clipboard operation
Megatron-LM copied to clipboard

Published 20 hours ago • verified publisher icon NVIDIA

Distributed Mamba Training

Open SkanderBS2024 opened this issue 7 months ago • 7 comments

How to customise the train.sh for a distributed Mamba Training ?

Hello, As i've seen in the megatron modules, there isn't a pre-defined bash script to pre-train a mamba model on multi-gpu, how can i set it up for model / data parallelism ...

SkanderBS2024 avatar Jul 23 '24 13:07 SkanderBS2024