wangyao

[Question] How does the gate coordinate across ranks in expert parallelism?

3

Hi, I'm trying to understand how the Gate module works in Tutel's MoE implementation. Since each rank only maintains a subset of experts (num_experts_per_device), but the Gate output seems to...