MoE-Mamba icon indicating copy to clipboard operation
MoE-Mamba copied to clipboard

Published 20 hours ago • verified publisher icon kyegomez

Is Class SwitchMixtureOfExperts unused in main model?

Open lunaaa95 opened this issue 7 months ago • 1 comments

I noticed that there are 2 version of class MoE in the repo. One is in model.py, named SwitchMoE, which is used in MambaMoE. While another MoE is in block.py, named SwitchMixtureOfExperts, which is not used in the model MambaMoE. Whats the purpose of that and whats the difference?

Upvote & Fund

  • We're using Polar.sh so you can upvote and help fund this issue.
  • We receive the funding once the issue is completed & confirmed by you.
  • Thank you in advance for helping prioritize & fund our backlog.
Fund with Polar

lunaaa95 avatar Jul 17 '24 11:07 lunaaa95