Question about multi-gate refer to multi-task learning

[Tokkiu]

Thanks for your contribution and excellent work of tutel! I am wondering can I use tutel to implement the multi-gate above experts like the picture as follows?

Currently, I can't see any similar solution in example files.

[ghostplant]

Do you mean something like this?

self._layer = tutel.moe.moe_layer(gate1_type={..}, gate2_type={..}, ..)

output1 = self._layer(data, use_gate1)
output2 = self._layer(data, use_gate2)

[Tokkiu]

@ghostplant Yes! And how to specify 'use_gate1' and 'use_gate2'?

[ghostplant]

We are going to merge this: https://github.com/microsoft/tutel/pull/71/files You can create new moe layers by specifying a list of original gating types. And when forwarding the moe layer, you can use self._moe_layer(data, gate_index=) to choose which gate to use.

[Tokkiu]

We are going to merge this: https://github.com/microsoft/tutel/pull/71/files You can create new moe layers by specifying a list of original gating types. And when forwarding the moe layer, you can use self._moe_layer(data, gate_index=) to choose which gate to use.

@ghostplant Nice work! Looking forward to your new feature.

[ghostplant]

It's done. Feel free to share any feedbacks. Thanks!