FasterMoE shadow expert implement

[Guodanding]

Hello, i want to implement FasterMoE shadow expert base on ColossalAI-MoeHybridParallel. is it possible? how can i achieve it?

[wangbluo]

Hi, of course it's possible, as FasterMoe also use Expert Parallel.

Thank you for your attention for ColossalAI, firstly you can fork a repository of ColossalAI, then you can write a demo, tested it locally, then raise a Pull Requests. We will do code review and feature tests, if all code tests is passed and reviewed, your code will be pushed into ColossalAI.