mergekit icon indicating copy to clipboard operation
mergekit copied to clipboard

Published 20 hours ago • verified publisher icon arcee-ai

#feature request# MoE structure activate expert number selection

Open Xingxiangrui opened this issue 1 year ago • 1 comments

Thanks for your wonderful job. Current mergekit-moe support merge experts and activate 2 of them. Can we change the number of activated experts ? such as activate 4 experts ?

Xingxiangrui avatar Jan 15 '24 02:01 Xingxiangrui

I have the same question. And I am trying to rewrite the mixtral_moe.py file.

ZhangEnmao avatar Jan 15 '24 10:01 ZhangEnmao

Good idea! In f98963bb7100ff523b108afc5f5ae9c4e85732eb I've added an experts_per_token option to the config files for mergekit-moe. If you leave it out it will default to two.

cg123 avatar Jan 16 '24 02:01 cg123