transformers
transformers copied to clipboard
[WIP]`NLLB-MoE` Adds the moe model
What does this PR do?
Fixes #21300 To-Dos:
- [x] Conversion script and original weights available here
- [x] Converted checkpoints and configuration file available: - moe-128 experts
- [x] Make the common tests go green
- [x] Implement top 2 gating mecanism
- [x] Add integration tests for:
- [x] the routers
- [x] the logits
- [x] the generation using greedy search
- [x] Cleanup the PR
The documentation is not available anymore as the PR was closed or merged.