When can we have a the MOE checkpoint convert script.

[shamanez]

As mentioned here, having a proper MOE/Mixtral checkpoint converter script will help us to fine-tune Mixtral.

[yqli2420]

+1

[hwdef]

I also strongly need this tool

https://github.com/NVIDIA/Megatron-LM/issues/756#issuecomment-2126186633

[oecompmind]

Here is some information about converting Huggingface checkpoints to Nemo. It seems there is a conversion script available on GitHub. Although I haven't confirmed it, it might be useful. https://medium.com/karakuri/train-moes-on-aws-trainium-a0ebb599fbda and https://github.com/abeja-inc/Megatron-LM

[yanring]

Sorry for the late response, please check: https://github.com/NVIDIA/Megatron-LM/tree/main/examples/mixtral

[yqli2420]

Sorry for the late response, please check: https://github.com/NVIDIA/Megatron-LM/tree/main/examples/mixtral

Does it support megatron_to_transformers? I am currently using https://github.com/alibaba/Pai-Megatron-Patch/blob/main/toolkits/model_checkpoints_convertor/mistral/hf2mcore.py

[github-actions[bot]]

Marking as stale. No activity in 60 days.