fastmoe icon indicating copy to clipboard operation
fastmoe copied to clipboard

Published 20 hours ago • verified publisher icon laekov

About switch_gate

Open Heihaierr opened this issue 11 months ago • 1 comments

Hi, I'm trying to implement a simpler version of switch transformer following your work. But the detail of switch_gate is invisible, like limit_by_capacity. My implementation has a slight different result with fastmoe.

Can you release the detail code of switch_gate?

Thanks.

Heihaierr avatar Mar 06 '24 08:03 Heihaierr

The pruning function is implemented here.

laekov avatar Mar 11 '24 06:03 laekov