MOE Support

[ziqipang]

Hi, I am very impressed by the project and have learned a lot! Just curious whether this minimal implementation plans to support MOE architectures recently? Thank you!

[DarkSharpness]

Hi. Really glad you found the project useful.

At the moment, this minimal implementation doesn't explicitly target MoE architectures. The optimization space in MoE is quite huge (EP, TP, fused moe kernel), and supporting it properly would add significant complexity to the codebase. Nevertheless, it's something very interesting and we could give it a try once the core pieces are more stable.

Feel free to open a PR or share a proposal here. Happy to discuss and iterate :)