Adding NLLB-200 - MoE - 54.5B for no language left behind

[PierreColombo]

System Info

Hello @LysandreJik, Thanks a lot for your work on no language left behind.

Is there any plan to add the 54.4B Model?

Kindest regards

Who can help?

No response

Information

• [ ] The official example scripts
• [ ] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [ ] My own task or dataset (give details below)

Reproduction

Improvement

Expected behavior

Improvement

[LysandreJik]

WDYT @ArthurZucker @younesbelkada given your work on MoEs?

[ArthurZucker]

Sure, we can add this to the to dos, @PierreColombo could you add the link to the open sourced checkpoints?

[PierreColombo]

Hi Thanks for your positive answer.

Code is here: https://github.com/facebookresearch/fairseq/tree/nllb

Checkpoints are here : https://tinyurl.com/nllb200moe54bmodel

Thanks !

[nunonmg]

Hi all, This would be greatly appreciated! Thanks

[julien-c]

also cc @sheonhan re. NNLB

[gsarti]

+1, would love to see it!

[BrightXiaoHan]

+1 here.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[julien-c]

unstale?

[PierreColombo]

We went for the fairseq implementation :'(

[sgugger]

Friendly ping @ArthurZucker

[ArthurZucker]

Yes! @sheonhan mentioned wanting to take this, otherwise will gladly sprint !

[sheonhan]

Since I'm working on the Image Completion Transformer at the moment, I might be blocking the folks who want to use it asap, so you should go ahead! @ArthurZucker