InternEvo
InternEvo copied to clipboard
fix(mlp): enhance mlp_layer_fusion
Motivation
- The
mlp_layer_fusionconfig is useful in MoE; therefore, a warning is added to recommend that users set this config to True in the MoE model. - When testing MoE model, the call of delattr in launch.py will raise the AttributeError even though the attribute exists. Therefore, add the delattr method in Config Class.
- Refactor the mlp's forward.
Modification
- internlm/core/context/parallel_context.py
- internlm/initialize/launch.py
- internlm/model/modules/mlp.py
Checklist
Before PR:
- [x] Pre-commit or other linting tools are used to fix the potential lint issues.