[FEATURE]: Expert Parallel for qwen/deepseek

[Guodanding]

Describe the feature

Hello, are there any existing implementations of expert parallel code for the new MoE model, like qwen and deepseek?

[shiyongde]

need FP8 training deepseek-MOE

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

need FP8 training deepseek-MOE

[ver217]

EP for Deepseek V3 is implemented, see our latest blog.

[ver217]

need FP8 training deepseek-MOE

FP8 gemm kernel released by deepseek github repo now is less efficient than BF16 gemm provided by cublas sometimes. We will release blockwise FP8 training feature until we resolve the efficiency issue.