OpenMoE icon indicating copy to clipboard operation
OpenMoE copied to clipboard

Published 20 hours ago • verified publisher icon XueFuzhao

tokens routing

Open wkml opened this issue 4 months ago • 6 comments

thanks for your work! It is very valuable! I would like to know how you got your conclusion about token routing, since input is affected by attention and rope, it is not logical that there should be a fixed routing for each token, how should I reproduce your result about this part?

wkml avatar Mar 04 '24 06:03 wkml